2023-07-12 00:31:36,726 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestZooKeeper timeout: 13 mins 2023-07-12 00:31:36,915 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09 2023-07-12 00:31:36,924 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/cluster_2bc97589-c177-c63b-7f92-2672baa4b2fe, deleteOnExit=true 2023-07-12 00:31:36,925 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/test.cache.data in system properties and HBase conf 2023-07-12 00:31:36,926 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/hadoop.tmp.dir in system properties and HBase conf 2023-07-12 00:31:36,926 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/hadoop.log.dir in system properties and HBase conf 2023-07-12 00:31:36,927 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-12 00:31:36,927 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-12 00:31:36,928 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-12 00:31:37,032 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-12 00:31:37,357 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-12 00:31:37,361 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-12 00:31:37,361 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-12 00:31:37,361 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-12 00:31:37,362 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 00:31:37,362 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-12 00:31:37,362 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-12 00:31:37,362 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 00:31:37,363 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 00:31:37,363 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-12 00:31:37,364 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/nfs.dump.dir in system properties and HBase conf 2023-07-12 00:31:37,364 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/java.io.tmpdir in system properties and HBase conf 2023-07-12 00:31:37,364 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 00:31:37,365 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-12 00:31:37,365 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-12 00:31:37,802 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 00:31:37,805 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 00:31:38,040 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-12 00:31:38,193 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-12 00:31:38,212 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 00:31:38,251 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 00:31:38,282 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/java.io.tmpdir/Jetty_localhost_localdomain_38283_hdfs____z1azgi/webapp 2023-07-12 00:31:38,416 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:38283 2023-07-12 00:31:38,424 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 00:31:38,424 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 00:31:38,765 WARN [Listener at localhost.localdomain/44599] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 00:31:38,864 WARN [Listener at localhost.localdomain/44599] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-12 00:31:38,881 WARN [Listener at localhost.localdomain/44599] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 00:31:38,888 INFO [Listener at localhost.localdomain/44599] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 00:31:38,893 INFO [Listener at localhost.localdomain/44599] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/java.io.tmpdir/Jetty_localhost_46357_datanode____wja7cq/webapp 2023-07-12 00:31:38,973 INFO [Listener at localhost.localdomain/44599] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46357 2023-07-12 00:31:39,267 WARN [Listener at localhost.localdomain/34031] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 00:31:39,276 WARN [Listener at localhost.localdomain/34031] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-12 00:31:39,280 WARN [Listener at localhost.localdomain/34031] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 00:31:39,282 INFO [Listener at localhost.localdomain/34031] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 00:31:39,287 INFO [Listener at localhost.localdomain/34031] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/java.io.tmpdir/Jetty_localhost_45287_datanode____.91ayyx/webapp 2023-07-12 00:31:39,373 INFO [Listener at localhost.localdomain/34031] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45287 2023-07-12 00:31:39,386 WARN [Listener at localhost.localdomain/33057] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 00:31:39,661 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x3d62dd025cba8740: Processing first storage report for DS-b30c5409-0068-4e2f-8885-a29cef6ccd7f from datanode b933a876-9f4f-46e9-9b59-29deef9caaf8 2023-07-12 00:31:39,663 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x3d62dd025cba8740: from storage DS-b30c5409-0068-4e2f-8885-a29cef6ccd7f node DatanodeRegistration(127.0.0.1:45447, datanodeUuid=b933a876-9f4f-46e9-9b59-29deef9caaf8, infoPort=34371, infoSecurePort=0, ipcPort=33057, storageInfo=lv=-57;cid=testClusterID;nsid=1155481977;c=1689121897866), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-12 00:31:39,663 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x11c5c69ad43d5e93: Processing first storage report for DS-6daa5430-1aef-49ba-8ebb-d10b83b7cb0f from datanode 6716b36f-abbe-41de-9ac2-625eebd1812d 2023-07-12 00:31:39,663 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x11c5c69ad43d5e93: from storage DS-6daa5430-1aef-49ba-8ebb-d10b83b7cb0f node DatanodeRegistration(127.0.0.1:36641, datanodeUuid=6716b36f-abbe-41de-9ac2-625eebd1812d, infoPort=34023, infoSecurePort=0, ipcPort=34031, storageInfo=lv=-57;cid=testClusterID;nsid=1155481977;c=1689121897866), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 00:31:39,663 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x3d62dd025cba8740: Processing first storage report for DS-5d643ca5-40a6-46de-9776-f26ff8eaabfb from datanode b933a876-9f4f-46e9-9b59-29deef9caaf8 2023-07-12 00:31:39,663 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x3d62dd025cba8740: from storage DS-5d643ca5-40a6-46de-9776-f26ff8eaabfb node DatanodeRegistration(127.0.0.1:45447, datanodeUuid=b933a876-9f4f-46e9-9b59-29deef9caaf8, infoPort=34371, infoSecurePort=0, ipcPort=33057, storageInfo=lv=-57;cid=testClusterID;nsid=1155481977;c=1689121897866), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-12 00:31:39,664 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x11c5c69ad43d5e93: Processing first storage report for DS-bede552e-e8d8-485e-a903-74475d6d7ee1 from datanode 6716b36f-abbe-41de-9ac2-625eebd1812d 2023-07-12 00:31:39,664 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x11c5c69ad43d5e93: from storage DS-bede552e-e8d8-485e-a903-74475d6d7ee1 node DatanodeRegistration(127.0.0.1:36641, datanodeUuid=6716b36f-abbe-41de-9ac2-625eebd1812d, infoPort=34023, infoSecurePort=0, ipcPort=34031, storageInfo=lv=-57;cid=testClusterID;nsid=1155481977;c=1689121897866), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 00:31:39,740 DEBUG [Listener at localhost.localdomain/33057] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09 2023-07-12 00:31:39,792 INFO [Listener at localhost.localdomain/33057] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/cluster_2bc97589-c177-c63b-7f92-2672baa4b2fe/zookeeper_0, clientPort=58965, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/cluster_2bc97589-c177-c63b-7f92-2672baa4b2fe/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/cluster_2bc97589-c177-c63b-7f92-2672baa4b2fe/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-12 00:31:39,805 INFO [Listener at localhost.localdomain/33057] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=58965 2023-07-12 00:31:39,807 INFO [Listener at localhost.localdomain/33057] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/test.cache.data in system properties and HBase conf 2023-07-12 00:31:39,808 INFO [Listener at localhost.localdomain/33057] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/hadoop.tmp.dir in system properties and HBase conf 2023-07-12 00:31:39,808 INFO [Listener at localhost.localdomain/33057] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/hadoop.log.dir in system properties and HBase conf 2023-07-12 00:31:39,808 INFO [Listener at localhost.localdomain/33057] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-12 00:31:39,808 INFO [Listener at localhost.localdomain/33057] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-12 00:31:39,809 INFO [Listener at localhost.localdomain/33057] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-12 00:31:39,818 INFO [Listener at localhost.localdomain/33057] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:31:39,821 INFO [Listener at localhost.localdomain/33057] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:31:39,834 INFO [Listener at localhost.localdomain/33057] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-12 00:31:39,834 INFO [Listener at localhost.localdomain/33057] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-12 00:31:39,834 INFO [Listener at localhost.localdomain/33057] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-12 00:31:39,835 INFO [Listener at localhost.localdomain/33057] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 00:31:39,835 INFO [Listener at localhost.localdomain/33057] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-12 00:31:39,835 INFO [Listener at localhost.localdomain/33057] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-12 00:31:39,835 INFO [Listener at localhost.localdomain/33057] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 00:31:39,835 INFO [Listener at localhost.localdomain/33057] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 00:31:39,836 INFO [Listener at localhost.localdomain/33057] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-12 00:31:39,836 INFO [Listener at localhost.localdomain/33057] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/nfs.dump.dir in system properties and HBase conf 2023-07-12 00:31:39,836 INFO [Listener at localhost.localdomain/33057] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/java.io.tmpdir in system properties and HBase conf 2023-07-12 00:31:39,836 INFO [Listener at localhost.localdomain/33057] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 00:31:39,837 INFO [Listener at localhost.localdomain/33057] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-12 00:31:39,837 INFO [Listener at localhost.localdomain/33057] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-12 00:31:39,841 WARN [Listener at localhost.localdomain/33057] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 00:31:39,841 WARN [Listener at localhost.localdomain/33057] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 00:31:39,869 WARN [Listener at localhost.localdomain/33057] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 00:31:39,873 INFO [Listener at localhost.localdomain/33057] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 00:31:39,881 INFO [Listener at localhost.localdomain/33057] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/java.io.tmpdir/Jetty_localhost_localdomain_45485_hdfs____.oerj12/webapp 2023-07-12 00:31:39,987 INFO [Listener at localhost.localdomain/33057] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:45485 2023-07-12 00:31:39,990 WARN [Listener at localhost.localdomain/33057] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 00:31:39,990 WARN [Listener at localhost.localdomain/33057] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 00:31:40,015 WARN [Listener at localhost.localdomain/44295] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 00:31:40,029 WARN [Listener at localhost.localdomain/44295] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-12 00:31:40,032 WARN [Listener at localhost.localdomain/44295] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 00:31:40,034 INFO [Listener at localhost.localdomain/44295] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 00:31:40,039 INFO [Listener at localhost.localdomain/44295] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/java.io.tmpdir/Jetty_localhost_37407_datanode____.2kz1oi/webapp 2023-07-12 00:31:40,117 INFO [Listener at localhost.localdomain/44295] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37407 2023-07-12 00:31:40,124 WARN [Listener at localhost.localdomain/34417] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 00:31:40,137 WARN [Listener at localhost.localdomain/34417] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-12 00:31:40,141 WARN [Listener at localhost.localdomain/34417] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 00:31:40,143 INFO [Listener at localhost.localdomain/34417] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 00:31:40,151 INFO [Listener at localhost.localdomain/34417] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/java.io.tmpdir/Jetty_localhost_34369_datanode____8z3ycc/webapp 2023-07-12 00:31:40,197 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x35b0f6b743a2fedf: Processing first storage report for DS-24609d7a-0366-47b4-b383-81fc18b4b2eb from datanode 55ed1f65-bd9a-4ab5-8a8a-ac56d85c09c2 2023-07-12 00:31:40,197 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x35b0f6b743a2fedf: from storage DS-24609d7a-0366-47b4-b383-81fc18b4b2eb node DatanodeRegistration(127.0.0.1:41473, datanodeUuid=55ed1f65-bd9a-4ab5-8a8a-ac56d85c09c2, infoPort=45403, infoSecurePort=0, ipcPort=34417, storageInfo=lv=-57;cid=testClusterID;nsid=765932182;c=1689121899843), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-12 00:31:40,198 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x35b0f6b743a2fedf: Processing first storage report for DS-7fe63423-4cb4-40b4-84b6-db8bed712c11 from datanode 55ed1f65-bd9a-4ab5-8a8a-ac56d85c09c2 2023-07-12 00:31:40,198 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x35b0f6b743a2fedf: from storage DS-7fe63423-4cb4-40b4-84b6-db8bed712c11 node DatanodeRegistration(127.0.0.1:41473, datanodeUuid=55ed1f65-bd9a-4ab5-8a8a-ac56d85c09c2, infoPort=45403, infoSecurePort=0, ipcPort=34417, storageInfo=lv=-57;cid=testClusterID;nsid=765932182;c=1689121899843), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 00:31:40,246 INFO [Listener at localhost.localdomain/34417] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34369 2023-07-12 00:31:40,266 WARN [Listener at localhost.localdomain/34481] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 00:31:40,348 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x49109d290be5213b: Processing first storage report for DS-21c4b7b1-1127-402f-b86f-1701bea8bc66 from datanode 6fb7e7dc-4325-40ec-b46c-1693148a1c05 2023-07-12 00:31:40,349 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x49109d290be5213b: from storage DS-21c4b7b1-1127-402f-b86f-1701bea8bc66 node DatanodeRegistration(127.0.0.1:43593, datanodeUuid=6fb7e7dc-4325-40ec-b46c-1693148a1c05, infoPort=32939, infoSecurePort=0, ipcPort=34481, storageInfo=lv=-57;cid=testClusterID;nsid=765932182;c=1689121899843), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-12 00:31:40,349 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x49109d290be5213b: Processing first storage report for DS-4c6a1a17-61d4-479e-af59-0b90af0d8cb8 from datanode 6fb7e7dc-4325-40ec-b46c-1693148a1c05 2023-07-12 00:31:40,349 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x49109d290be5213b: from storage DS-4c6a1a17-61d4-479e-af59-0b90af0d8cb8 node DatanodeRegistration(127.0.0.1:43593, datanodeUuid=6fb7e7dc-4325-40ec-b46c-1693148a1c05, infoPort=32939, infoSecurePort=0, ipcPort=34481, storageInfo=lv=-57;cid=testClusterID;nsid=765932182;c=1689121899843), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 00:31:40,381 DEBUG [Listener at localhost.localdomain/34481] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09 2023-07-12 00:31:40,399 INFO [Listener at localhost.localdomain/34481] hbase.ResourceChecker(147): before: TestZooKeeper#testRegionServerSessionExpired Thread=208, OpenFileDescriptor=471, MaxFileDescriptor=60000, SystemLoadAverage=163, ProcessCount=173, AvailableMemoryMB=5182 2023-07-12 00:31:40,402 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:31:40,404 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:31:41,072 INFO [Listener at localhost.localdomain/34481] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983 with version=8 2023-07-12 00:31:41,072 INFO [Listener at localhost.localdomain/34481] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/hbase-staging 2023-07-12 00:31:41,343 INFO [Listener at localhost.localdomain/34481] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-12 00:31:41,878 INFO [Listener at localhost.localdomain/34481] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-07-12 00:31:41,919 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:31:41,920 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 00:31:41,920 INFO [Listener at localhost.localdomain/34481] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 00:31:41,921 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:31:41,921 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 00:31:42,101 INFO [Listener at localhost.localdomain/34481] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 00:31:42,175 DEBUG [Listener at localhost.localdomain/34481] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-12 00:31:42,254 INFO [Listener at localhost.localdomain/34481] ipc.NettyRpcServer(120): Bind to /148.251.75.209:35377 2023-07-12 00:31:42,264 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:31:42,266 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:31:42,284 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=master:35377 connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:31:42,315 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:353770x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:31:42,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:35377-0x1007e9741810000 connected 2023-07-12 00:31:42,344 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:31:42,345 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 00:31:42,348 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 00:31:42,355 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35377 2023-07-12 00:31:42,356 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35377 2023-07-12 00:31:42,356 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35377 2023-07-12 00:31:42,356 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35377 2023-07-12 00:31:42,356 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35377 2023-07-12 00:31:42,361 INFO [Listener at localhost.localdomain/34481] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983, hbase.cluster.distributed=false 2023-07-12 00:31:42,394 INFO [Listener at localhost.localdomain/34481] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-07-12 00:31:42,394 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:31:42,394 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 00:31:42,395 INFO [Listener at localhost.localdomain/34481] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 00:31:42,395 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:31:42,395 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 00:31:42,395 INFO [Listener at localhost.localdomain/34481] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 00:31:42,397 INFO [Listener at localhost.localdomain/34481] ipc.NettyRpcServer(120): Bind to /148.251.75.209:40891 2023-07-12 00:31:42,397 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:31:42,399 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:31:42,400 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=master:40891 connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:31:42,430 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:408910x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:31:42,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:40891-0x1007e9741810001 connected 2023-07-12 00:31:42,444 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:31:42,445 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 00:31:42,446 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 00:31:42,446 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40891 2023-07-12 00:31:42,447 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40891 2023-07-12 00:31:42,447 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40891 2023-07-12 00:31:42,447 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40891 2023-07-12 00:31:42,448 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40891 2023-07-12 00:31:42,448 INFO [Listener at localhost.localdomain/34481] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983, hbase.cluster.distributed=false 2023-07-12 00:31:42,478 INFO [Listener at localhost.localdomain/34481] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-07-12 00:31:42,478 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:31:42,478 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 00:31:42,478 INFO [Listener at localhost.localdomain/34481] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 00:31:42,478 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:31:42,479 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 00:31:42,484 INFO [Listener at localhost.localdomain/34481] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 00:31:42,487 INFO [Listener at localhost.localdomain/34481] ipc.NettyRpcServer(120): Bind to /148.251.75.209:37983 2023-07-12 00:31:42,490 INFO [Listener at localhost.localdomain/34481] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 00:31:42,498 DEBUG [Listener at localhost.localdomain/34481] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 00:31:42,500 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:31:42,502 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:31:42,504 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:37983 connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:31:42,509 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:379830x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:31:42,510 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:379830x0, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:31:42,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:37983-0x1007e9741810002 connected 2023-07-12 00:31:42,513 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:37983-0x1007e9741810002, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 00:31:42,514 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:37983-0x1007e9741810002, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 00:31:42,516 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37983 2023-07-12 00:31:42,517 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37983 2023-07-12 00:31:42,517 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37983 2023-07-12 00:31:42,517 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37983 2023-07-12 00:31:42,518 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37983 2023-07-12 00:31:42,528 INFO [Listener at localhost.localdomain/34481] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-07-12 00:31:42,528 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:31:42,529 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 00:31:42,529 INFO [Listener at localhost.localdomain/34481] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 00:31:42,529 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:31:42,529 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 00:31:42,529 INFO [Listener at localhost.localdomain/34481] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 00:31:42,531 INFO [Listener at localhost.localdomain/34481] ipc.NettyRpcServer(120): Bind to /148.251.75.209:42559 2023-07-12 00:31:42,532 INFO [Listener at localhost.localdomain/34481] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 00:31:42,533 DEBUG [Listener at localhost.localdomain/34481] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 00:31:42,534 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:31:42,535 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:31:42,536 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:42559 connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:31:42,540 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:425590x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:31:42,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:42559-0x1007e9741810003 connected 2023-07-12 00:31:42,542 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:42559-0x1007e9741810003, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:31:42,543 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:42559-0x1007e9741810003, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 00:31:42,544 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:42559-0x1007e9741810003, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 00:31:42,544 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42559 2023-07-12 00:31:42,544 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42559 2023-07-12 00:31:42,545 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42559 2023-07-12 00:31:42,545 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42559 2023-07-12 00:31:42,545 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42559 2023-07-12 00:31:42,547 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,35377,1689121901202 2023-07-12 00:31:42,548 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,40891,1689121902393 2023-07-12 00:31:42,556 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:31:42,556 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:31:42,559 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,40891,1689121902393 2023-07-12 00:31:42,559 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,35377,1689121901202 2023-07-12 00:31:42,583 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:31:42,583 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:31:42,583 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:42559-0x1007e9741810003, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:31:42,583 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:37983-0x1007e9741810002, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:31:42,583 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:42,583 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:42,584 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:31:42,585 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:31:42,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:31:42,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:31:42,586 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,40891,1689121902393 from backup master directory 2023-07-12 00:31:42,587 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(278): Another master is the active master, jenkins-hbase20.apache.org,40891,1689121902393; waiting to become the next active master 2023-07-12 00:31:42,598 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:31:42,598 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,40891,1689121902393 2023-07-12 00:31:42,598 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:31:42,599 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 00:31:42,599 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,40891,1689121902393 2023-07-12 00:31:42,604 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-12 00:31:42,606 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-12 00:31:42,698 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/hbase.id with ID: b94c042f-d510-47b7-9d4c-011dde64d627 2023-07-12 00:31:42,746 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:31:42,761 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:42,761 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:42,807 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x056bd26c to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:31:42,840 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59587153, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:31:42,867 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 00:31:42,869 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 00:31:42,887 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-12 00:31:42,887 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-12 00:31:42,889 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:31:42,892 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:31:42,893 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:31:42,920 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store-tmp 2023-07-12 00:31:42,952 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:31:42,953 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 00:31:42,953 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:31:42,953 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:31:42,953 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 00:31:42,953 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:31:42,953 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:31:42,953 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 00:31:42,955 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,40891,1689121902393 2023-07-12 00:31:42,976 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C40891%2C1689121902393, suffix=, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,40891,1689121902393, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/oldWALs, maxLogs=10 2023-07-12 00:31:43,028 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:31:43,028 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:31:43,035 DEBUG [RS-EventLoopGroup-5-3] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:31:43,099 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,40891,1689121902393/jenkins-hbase20.apache.org%2C40891%2C1689121902393.1689121902984 2023-07-12 00:31:43,100 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK], DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK]] 2023-07-12 00:31:43,101 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 00:31:43,101 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:31:43,105 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:31:43,106 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:31:43,170 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:31:43,177 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 00:31:43,203 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 00:31:43,216 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:31:43,224 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:31:43,226 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:31:43,250 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:31:43,257 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:31:43,259 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71174504, jitterRate=0.060582756996154785}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:31:43,259 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 00:31:43,260 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 00:31:43,280 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-12 00:31:43,280 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-12 00:31:43,283 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 00:31:43,286 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-12 00:31:43,320 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 34 msec 2023-07-12 00:31:43,321 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 00:31:43,351 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-12 00:31:43,357 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-12 00:31:43,387 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 00:31:43,388 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 00:31:43,393 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 00:31:43,396 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 00:31:43,398 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:43,398 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:43,399 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 00:31:43,400 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 00:31:43,410 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 00:31:43,413 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:42559-0x1007e9741810003, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 00:31:43,414 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 00:31:43,414 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 00:31:43,414 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:37983-0x1007e9741810002, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 00:31:43,414 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:43,414 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:43,414 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,40891,1689121902393, sessionid=0x1007e9741810001, setting cluster-up flag (Was=false) 2023-07-12 00:31:43,418 INFO [M:0;jenkins-hbase20:35377] regionserver.HRegionServer(951): ClusterId : b94c042f-d510-47b7-9d4c-011dde64d627 2023-07-12 00:31:43,429 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:43,429 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:43,433 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 00:31:43,434 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,40891,1689121902393 2023-07-12 00:31:43,437 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:43,437 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:43,440 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 00:31:43,441 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,40891,1689121902393 2023-07-12 00:31:43,444 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.hbase-snapshot/.tmp 2023-07-12 00:31:43,450 INFO [RS:0;jenkins-hbase20:37983] regionserver.HRegionServer(951): ClusterId : b94c042f-d510-47b7-9d4c-011dde64d627 2023-07-12 00:31:43,450 INFO [RS:1;jenkins-hbase20:42559] regionserver.HRegionServer(951): ClusterId : b94c042f-d510-47b7-9d4c-011dde64d627 2023-07-12 00:31:43,457 DEBUG [RS:1;jenkins-hbase20:42559] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 00:31:43,457 DEBUG [RS:0;jenkins-hbase20:37983] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 00:31:43,464 DEBUG [RS:0;jenkins-hbase20:37983] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 00:31:43,464 DEBUG [RS:1;jenkins-hbase20:42559] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 00:31:43,464 DEBUG [RS:0;jenkins-hbase20:37983] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 00:31:43,464 DEBUG [RS:1;jenkins-hbase20:42559] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 00:31:43,467 DEBUG [RS:1;jenkins-hbase20:42559] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 00:31:43,467 DEBUG [RS:0;jenkins-hbase20:37983] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 00:31:43,469 DEBUG [RS:1;jenkins-hbase20:42559] zookeeper.ReadOnlyZKClient(139): Connect 0x6f14fd09 to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:31:43,470 DEBUG [RS:0;jenkins-hbase20:37983] zookeeper.ReadOnlyZKClient(139): Connect 0x209b501c to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:31:43,482 DEBUG [RS:0;jenkins-hbase20:37983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2e5cba7f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:31:43,482 DEBUG [RS:1;jenkins-hbase20:42559] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@61d31e9b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:31:43,483 DEBUG [RS:0;jenkins-hbase20:37983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2a633b6c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-12 00:31:43,483 DEBUG [RS:1;jenkins-hbase20:42559] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65ec9679, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-12 00:31:43,510 DEBUG [RS:0;jenkins-hbase20:37983] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:37983 2023-07-12 00:31:43,511 DEBUG [RS:1;jenkins-hbase20:42559] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase20:42559 2023-07-12 00:31:43,515 INFO [RS:0;jenkins-hbase20:37983] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 00:31:43,516 INFO [RS:0;jenkins-hbase20:37983] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 00:31:43,515 INFO [RS:1;jenkins-hbase20:42559] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 00:31:43,516 INFO [RS:1;jenkins-hbase20:42559] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 00:31:43,516 DEBUG [RS:1;jenkins-hbase20:42559] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 00:31:43,516 DEBUG [RS:0;jenkins-hbase20:37983] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 00:31:43,519 INFO [RS:0;jenkins-hbase20:37983] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase20.apache.org,40891,1689121902393 with isa=jenkins-hbase20.apache.org/148.251.75.209:37983, startcode=1689121902477 2023-07-12 00:31:43,519 INFO [RS:1;jenkins-hbase20:42559] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase20.apache.org,40891,1689121902393 with isa=jenkins-hbase20.apache.org/148.251.75.209:42559, startcode=1689121902528 2023-07-12 00:31:43,534 DEBUG [RS:0;jenkins-hbase20:37983] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 00:31:43,534 DEBUG [RS:1;jenkins-hbase20:42559] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 00:31:43,580 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-12 00:31:43,588 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49053, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 00:31:43,588 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:33257, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 00:31:43,589 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:31:43,590 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:31:43,590 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:31:43,590 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:31:43,590 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-07-12 00:31:43,590 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:43,590 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-12 00:31:43,590 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:43,595 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689121933595 2023-07-12 00:31:43,595 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40891] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:579) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:31:43,598 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 00:31:43,600 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 00:31:43,601 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-12 00:31:43,606 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40891] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:579) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:31:43,608 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 00:31:43,610 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 00:31:43,624 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 00:31:43,625 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 00:31:43,625 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 00:31:43,625 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 00:31:43,626 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:43,627 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 00:31:43,628 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 00:31:43,629 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 00:31:43,630 DEBUG [RS:1;jenkins-hbase20:42559] regionserver.HRegionServer(2830): Master is not running yet 2023-07-12 00:31:43,630 DEBUG [RS:0;jenkins-hbase20:37983] regionserver.HRegionServer(2830): Master is not running yet 2023-07-12 00:31:43,632 WARN [RS:1;jenkins-hbase20:42559] regionserver.HRegionServer(1030): reportForDuty failed; sleeping 100 ms and then retrying. 2023-07-12 00:31:43,632 WARN [RS:0;jenkins-hbase20:37983] regionserver.HRegionServer(1030): reportForDuty failed; sleeping 100 ms and then retrying. 2023-07-12 00:31:43,633 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 00:31:43,634 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 00:31:43,637 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689121903635,5,FailOnTimeoutGroup] 2023-07-12 00:31:43,638 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689121903637,5,FailOnTimeoutGroup] 2023-07-12 00:31:43,638 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:43,638 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 00:31:43,640 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:43,640 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:43,659 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 00:31:43,661 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 00:31:43,661 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983 2023-07-12 00:31:43,690 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:31:43,694 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 00:31:43,700 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/info 2023-07-12 00:31:43,701 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 00:31:43,702 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:31:43,702 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 00:31:43,705 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/rep_barrier 2023-07-12 00:31:43,705 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 00:31:43,706 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:31:43,706 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 00:31:43,708 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/table 2023-07-12 00:31:43,709 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 00:31:43,710 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:31:43,711 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740 2023-07-12 00:31:43,712 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740 2023-07-12 00:31:43,716 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 00:31:43,718 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 00:31:43,722 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:31:43,723 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67803404, jitterRate=0.010349452495574951}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 00:31:43,723 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 00:31:43,723 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 00:31:43,723 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 00:31:43,723 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 00:31:43,723 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 00:31:43,723 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 00:31:43,724 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 00:31:43,725 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 00:31:43,729 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 00:31:43,730 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-12 00:31:43,732 INFO [RS:0;jenkins-hbase20:37983] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase20.apache.org,40891,1689121902393 with isa=jenkins-hbase20.apache.org/148.251.75.209:37983, startcode=1689121902477 2023-07-12 00:31:43,733 INFO [RS:1;jenkins-hbase20:42559] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase20.apache.org,40891,1689121902393 with isa=jenkins-hbase20.apache.org/148.251.75.209:42559, startcode=1689121902528 2023-07-12 00:31:43,739 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40891] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,42559,1689121902528 2023-07-12 00:31:43,739 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-12 00:31:43,744 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40891] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,37983,1689121902477 2023-07-12 00:31:43,745 DEBUG [RS:1;jenkins-hbase20:42559] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983 2023-07-12 00:31:43,746 DEBUG [RS:1;jenkins-hbase20:42559] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:44295 2023-07-12 00:31:43,746 DEBUG [RS:1;jenkins-hbase20:42559] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 00:31:43,746 DEBUG [RS:0;jenkins-hbase20:37983] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983 2023-07-12 00:31:43,746 DEBUG [RS:0;jenkins-hbase20:37983] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:44295 2023-07-12 00:31:43,746 DEBUG [RS:0;jenkins-hbase20:37983] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 00:31:43,753 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:31:43,753 DEBUG [RS:1;jenkins-hbase20:42559] zookeeper.ZKUtil(162): regionserver:42559-0x1007e9741810003, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,42559,1689121902528 2023-07-12 00:31:43,753 DEBUG [RS:0;jenkins-hbase20:37983] zookeeper.ZKUtil(162): regionserver:37983-0x1007e9741810002, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,37983,1689121902477 2023-07-12 00:31:43,754 WARN [RS:1;jenkins-hbase20:42559] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 00:31:43,754 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-12 00:31:43,754 WARN [RS:0;jenkins-hbase20:37983] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 00:31:43,755 INFO [RS:1;jenkins-hbase20:42559] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:31:43,755 DEBUG [RS:1;jenkins-hbase20:42559] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,42559,1689121902528 2023-07-12 00:31:43,755 INFO [RS:0;jenkins-hbase20:37983] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:31:43,756 DEBUG [RS:0;jenkins-hbase20:37983] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477 2023-07-12 00:31:43,757 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,42559,1689121902528] 2023-07-12 00:31:43,758 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,37983,1689121902477] 2023-07-12 00:31:43,762 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-12 00:31:43,769 DEBUG [RS:0;jenkins-hbase20:37983] zookeeper.ZKUtil(162): regionserver:37983-0x1007e9741810002, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,37983,1689121902477 2023-07-12 00:31:43,769 DEBUG [RS:1;jenkins-hbase20:42559] zookeeper.ZKUtil(162): regionserver:42559-0x1007e9741810003, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,37983,1689121902477 2023-07-12 00:31:43,769 DEBUG [RS:0;jenkins-hbase20:37983] zookeeper.ZKUtil(162): regionserver:37983-0x1007e9741810002, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,42559,1689121902528 2023-07-12 00:31:43,769 DEBUG [RS:1;jenkins-hbase20:42559] zookeeper.ZKUtil(162): regionserver:42559-0x1007e9741810003, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,42559,1689121902528 2023-07-12 00:31:43,779 DEBUG [RS:1;jenkins-hbase20:42559] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 00:31:43,779 DEBUG [RS:0;jenkins-hbase20:37983] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 00:31:43,787 INFO [RS:1;jenkins-hbase20:42559] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 00:31:43,787 INFO [RS:0;jenkins-hbase20:37983] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 00:31:43,806 INFO [RS:0;jenkins-hbase20:37983] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 00:31:43,806 INFO [RS:1;jenkins-hbase20:42559] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 00:31:43,810 INFO [RS:0;jenkins-hbase20:37983] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 00:31:43,810 INFO [RS:1;jenkins-hbase20:42559] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 00:31:43,811 INFO [RS:0;jenkins-hbase20:37983] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:43,811 INFO [RS:1;jenkins-hbase20:42559] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:43,812 INFO [RS:0;jenkins-hbase20:37983] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 00:31:43,812 INFO [RS:1;jenkins-hbase20:42559] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 00:31:43,818 INFO [RS:1;jenkins-hbase20:42559] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:43,818 INFO [RS:0;jenkins-hbase20:37983] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:43,819 DEBUG [RS:1;jenkins-hbase20:42559] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:43,819 DEBUG [RS:0;jenkins-hbase20:37983] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:43,819 DEBUG [RS:1;jenkins-hbase20:42559] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:43,819 DEBUG [RS:0;jenkins-hbase20:37983] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:43,819 DEBUG [RS:1;jenkins-hbase20:42559] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:43,819 DEBUG [RS:0;jenkins-hbase20:37983] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:43,819 DEBUG [RS:1;jenkins-hbase20:42559] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:43,820 DEBUG [RS:0;jenkins-hbase20:37983] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:43,820 DEBUG [RS:1;jenkins-hbase20:42559] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:43,820 DEBUG [RS:0;jenkins-hbase20:37983] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:43,820 DEBUG [RS:1;jenkins-hbase20:42559] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-12 00:31:43,820 DEBUG [RS:0;jenkins-hbase20:37983] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-12 00:31:43,820 DEBUG [RS:1;jenkins-hbase20:42559] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:43,820 DEBUG [RS:0;jenkins-hbase20:37983] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:43,820 DEBUG [RS:1;jenkins-hbase20:42559] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:43,820 DEBUG [RS:0;jenkins-hbase20:37983] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:43,820 DEBUG [RS:1;jenkins-hbase20:42559] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:43,820 DEBUG [RS:0;jenkins-hbase20:37983] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:43,820 DEBUG [RS:1;jenkins-hbase20:42559] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:43,820 DEBUG [RS:0;jenkins-hbase20:37983] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:43,822 INFO [RS:1;jenkins-hbase20:42559] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:43,822 INFO [RS:0;jenkins-hbase20:37983] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:43,822 INFO [RS:1;jenkins-hbase20:42559] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:43,822 INFO [RS:0;jenkins-hbase20:37983] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:43,822 INFO [RS:1;jenkins-hbase20:42559] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:43,822 INFO [RS:0;jenkins-hbase20:37983] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:43,837 INFO [RS:0;jenkins-hbase20:37983] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 00:31:43,837 INFO [RS:1;jenkins-hbase20:42559] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 00:31:43,841 INFO [RS:0;jenkins-hbase20:37983] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,37983,1689121902477-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:43,841 INFO [RS:1;jenkins-hbase20:42559] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,42559,1689121902528-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:43,863 INFO [RS:0;jenkins-hbase20:37983] regionserver.Replication(203): jenkins-hbase20.apache.org,37983,1689121902477 started 2023-07-12 00:31:43,863 INFO [RS:1;jenkins-hbase20:42559] regionserver.Replication(203): jenkins-hbase20.apache.org,42559,1689121902528 started 2023-07-12 00:31:43,864 INFO [RS:0;jenkins-hbase20:37983] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,37983,1689121902477, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:37983, sessionid=0x1007e9741810002 2023-07-12 00:31:43,864 INFO [RS:1;jenkins-hbase20:42559] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,42559,1689121902528, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:42559, sessionid=0x1007e9741810003 2023-07-12 00:31:43,864 DEBUG [RS:0;jenkins-hbase20:37983] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 00:31:43,864 DEBUG [RS:1;jenkins-hbase20:42559] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 00:31:43,864 DEBUG [RS:0;jenkins-hbase20:37983] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,37983,1689121902477 2023-07-12 00:31:43,864 DEBUG [RS:1;jenkins-hbase20:42559] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,42559,1689121902528 2023-07-12 00:31:43,865 DEBUG [RS:0;jenkins-hbase20:37983] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,37983,1689121902477' 2023-07-12 00:31:43,865 DEBUG [RS:1;jenkins-hbase20:42559] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,42559,1689121902528' 2023-07-12 00:31:43,866 DEBUG [RS:1;jenkins-hbase20:42559] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 00:31:43,865 DEBUG [RS:0;jenkins-hbase20:37983] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 00:31:43,866 DEBUG [RS:1;jenkins-hbase20:42559] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 00:31:43,866 DEBUG [RS:0;jenkins-hbase20:37983] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 00:31:43,867 DEBUG [RS:0;jenkins-hbase20:37983] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 00:31:43,867 DEBUG [RS:1;jenkins-hbase20:42559] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 00:31:43,867 DEBUG [RS:0;jenkins-hbase20:37983] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 00:31:43,867 DEBUG [RS:1;jenkins-hbase20:42559] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 00:31:43,867 DEBUG [RS:0;jenkins-hbase20:37983] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,37983,1689121902477 2023-07-12 00:31:43,867 DEBUG [RS:1;jenkins-hbase20:42559] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,42559,1689121902528 2023-07-12 00:31:43,868 DEBUG [RS:0;jenkins-hbase20:37983] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,37983,1689121902477' 2023-07-12 00:31:43,868 DEBUG [RS:0;jenkins-hbase20:37983] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 00:31:43,868 DEBUG [RS:1;jenkins-hbase20:42559] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,42559,1689121902528' 2023-07-12 00:31:43,868 DEBUG [RS:1;jenkins-hbase20:42559] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 00:31:43,868 DEBUG [RS:0;jenkins-hbase20:37983] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 00:31:43,869 DEBUG [RS:1;jenkins-hbase20:42559] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 00:31:43,869 DEBUG [RS:0;jenkins-hbase20:37983] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 00:31:43,869 INFO [RS:0;jenkins-hbase20:37983] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 00:31:43,869 DEBUG [RS:1;jenkins-hbase20:42559] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 00:31:43,869 INFO [RS:0;jenkins-hbase20:37983] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 00:31:43,869 INFO [RS:1;jenkins-hbase20:42559] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 00:31:43,870 INFO [RS:1;jenkins-hbase20:42559] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 00:31:43,915 DEBUG [jenkins-hbase20:40891] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-12 00:31:43,918 DEBUG [jenkins-hbase20:40891] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-07-12 00:31:43,924 DEBUG [jenkins-hbase20:40891] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 00:31:43,924 DEBUG [jenkins-hbase20:40891] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 00:31:43,924 DEBUG [jenkins-hbase20:40891] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 00:31:43,927 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,37983,1689121902477, state=OPENING 2023-07-12 00:31:43,932 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-12 00:31:43,933 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:43,933 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:43,934 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 00:31:43,934 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 00:31:43,938 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,37983,1689121902477}] 2023-07-12 00:31:43,983 INFO [RS:1;jenkins-hbase20:42559] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C42559%2C1689121902528, suffix=, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,42559,1689121902528, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/oldWALs, maxLogs=32 2023-07-12 00:31:43,983 INFO [RS:0;jenkins-hbase20:37983] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C37983%2C1689121902477, suffix=, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/oldWALs, maxLogs=32 2023-07-12 00:31:44,006 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:31:44,006 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:31:44,007 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:31:44,008 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:31:44,019 INFO [RS:0;jenkins-hbase20:37983] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477/jenkins-hbase20.apache.org%2C37983%2C1689121902477.1689121903986 2023-07-12 00:31:44,019 INFO [RS:1;jenkins-hbase20:42559] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,42559,1689121902528/jenkins-hbase20.apache.org%2C42559%2C1689121902528.1689121903986 2023-07-12 00:31:44,019 DEBUG [RS:0;jenkins-hbase20:37983] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK], DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK]] 2023-07-12 00:31:44,020 DEBUG [RS:1;jenkins-hbase20:42559] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK], DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK]] 2023-07-12 00:31:44,126 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,37983,1689121902477 2023-07-12 00:31:44,128 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 00:31:44,132 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:58888, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 00:31:44,148 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-12 00:31:44,149 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:31:44,153 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/oldWALs, maxLogs=32 2023-07-12 00:31:44,170 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:31:44,170 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:31:44,176 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477/jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta 2023-07-12 00:31:44,176 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK], DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK]] 2023-07-12 00:31:44,177 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-12 00:31:44,179 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-12 00:31:44,193 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-12 00:31:44,197 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-12 00:31:44,201 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-12 00:31:44,202 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:31:44,202 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-12 00:31:44,202 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-12 00:31:44,205 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 00:31:44,207 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/info 2023-07-12 00:31:44,207 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/info 2023-07-12 00:31:44,208 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 00:31:44,210 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:31:44,210 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 00:31:44,212 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/rep_barrier 2023-07-12 00:31:44,212 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/rep_barrier 2023-07-12 00:31:44,212 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 00:31:44,213 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:31:44,213 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 00:31:44,215 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/table 2023-07-12 00:31:44,215 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/table 2023-07-12 00:31:44,215 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 00:31:44,216 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:31:44,218 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740 2023-07-12 00:31:44,222 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740 2023-07-12 00:31:44,225 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 00:31:44,228 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 00:31:44,230 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71121422, jitterRate=0.059791773557662964}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 00:31:44,230 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 00:31:44,239 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689121904120 2023-07-12 00:31:44,260 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-12 00:31:44,261 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-12 00:31:44,262 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,37983,1689121902477, state=OPEN 2023-07-12 00:31:44,264 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 00:31:44,264 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 00:31:44,264 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 00:31:44,264 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 00:31:44,269 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-12 00:31:44,269 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,37983,1689121902477 in 326 msec 2023-07-12 00:31:44,274 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-12 00:31:44,275 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 531 msec 2023-07-12 00:31:44,279 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 780 msec 2023-07-12 00:31:44,280 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689121904279, completionTime=-1 2023-07-12 00:31:44,280 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-12 00:31:44,280 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 00:31:44,334 DEBUG [hconnection-0x7d623b3f-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 00:31:44,338 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:58900, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 00:31:44,355 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-12 00:31:44,355 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689121964355 2023-07-12 00:31:44,355 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689122024355 2023-07-12 00:31:44,355 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 75 msec 2023-07-12 00:31:44,383 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,40891,1689121902393-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:44,383 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,40891,1689121902393-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:44,383 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,40891,1689121902393-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:44,385 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:40891, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:44,385 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:44,392 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 00:31:44,400 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-12 00:31:44,401 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 00:31:44,409 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-12 00:31:44,412 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 00:31:44,414 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 00:31:44,437 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011 2023-07-12 00:31:44,439 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011 empty. 2023-07-12 00:31:44,440 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011 2023-07-12 00:31:44,440 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-12 00:31:44,479 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-12 00:31:44,482 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 150e8dd2bd05831664bbcb60e0b3f011, NAME => 'hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp 2023-07-12 00:31:44,505 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:31:44,506 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 150e8dd2bd05831664bbcb60e0b3f011, disabling compactions & flushes 2023-07-12 00:31:44,506 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011. 2023-07-12 00:31:44,506 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011. 2023-07-12 00:31:44,506 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011. after waiting 0 ms 2023-07-12 00:31:44,506 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011. 2023-07-12 00:31:44,506 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011. 2023-07-12 00:31:44,506 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 150e8dd2bd05831664bbcb60e0b3f011: 2023-07-12 00:31:44,512 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 00:31:44,528 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689121904515"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689121904515"}]},"ts":"1689121904515"} 2023-07-12 00:31:44,553 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 00:31:44,556 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 00:31:44,561 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689121904556"}]},"ts":"1689121904556"} 2023-07-12 00:31:44,565 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-12 00:31:44,569 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-07-12 00:31:44,570 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 00:31:44,571 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 00:31:44,571 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 00:31:44,574 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=150e8dd2bd05831664bbcb60e0b3f011, ASSIGN}] 2023-07-12 00:31:44,578 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=150e8dd2bd05831664bbcb60e0b3f011, ASSIGN 2023-07-12 00:31:44,580 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=150e8dd2bd05831664bbcb60e0b3f011, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,37983,1689121902477; forceNewPlan=false, retain=false 2023-07-12 00:31:44,736 INFO [jenkins-hbase20:40891] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-12 00:31:44,737 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=150e8dd2bd05831664bbcb60e0b3f011, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,37983,1689121902477 2023-07-12 00:31:44,738 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689121904737"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121904737"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121904737"}]},"ts":"1689121904737"} 2023-07-12 00:31:44,746 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 150e8dd2bd05831664bbcb60e0b3f011, server=jenkins-hbase20.apache.org,37983,1689121902477}] 2023-07-12 00:31:44,912 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011. 2023-07-12 00:31:44,913 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 150e8dd2bd05831664bbcb60e0b3f011, NAME => 'hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011.', STARTKEY => '', ENDKEY => ''} 2023-07-12 00:31:44,915 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 150e8dd2bd05831664bbcb60e0b3f011 2023-07-12 00:31:44,915 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:31:44,915 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 150e8dd2bd05831664bbcb60e0b3f011 2023-07-12 00:31:44,915 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 150e8dd2bd05831664bbcb60e0b3f011 2023-07-12 00:31:44,917 INFO [StoreOpener-150e8dd2bd05831664bbcb60e0b3f011-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 150e8dd2bd05831664bbcb60e0b3f011 2023-07-12 00:31:44,919 DEBUG [StoreOpener-150e8dd2bd05831664bbcb60e0b3f011-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011/info 2023-07-12 00:31:44,919 DEBUG [StoreOpener-150e8dd2bd05831664bbcb60e0b3f011-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011/info 2023-07-12 00:31:44,920 INFO [StoreOpener-150e8dd2bd05831664bbcb60e0b3f011-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 150e8dd2bd05831664bbcb60e0b3f011 columnFamilyName info 2023-07-12 00:31:44,921 INFO [StoreOpener-150e8dd2bd05831664bbcb60e0b3f011-1] regionserver.HStore(310): Store=150e8dd2bd05831664bbcb60e0b3f011/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:31:44,923 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011 2023-07-12 00:31:44,924 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011 2023-07-12 00:31:44,928 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 150e8dd2bd05831664bbcb60e0b3f011 2023-07-12 00:31:44,931 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:31:44,932 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 150e8dd2bd05831664bbcb60e0b3f011; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60095672, jitterRate=-0.10450470447540283}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:31:44,932 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 150e8dd2bd05831664bbcb60e0b3f011: 2023-07-12 00:31:44,934 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011., pid=6, masterSystemTime=1689121904903 2023-07-12 00:31:44,939 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011. 2023-07-12 00:31:44,939 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011. 2023-07-12 00:31:44,941 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=150e8dd2bd05831664bbcb60e0b3f011, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,37983,1689121902477 2023-07-12 00:31:44,941 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689121904940"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121904940"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121904940"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121904940"}]},"ts":"1689121904940"} 2023-07-12 00:31:44,950 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-12 00:31:44,951 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 150e8dd2bd05831664bbcb60e0b3f011, server=jenkins-hbase20.apache.org,37983,1689121902477 in 201 msec 2023-07-12 00:31:44,954 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-12 00:31:44,954 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=150e8dd2bd05831664bbcb60e0b3f011, ASSIGN in 377 msec 2023-07-12 00:31:44,956 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 00:31:44,956 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689121904956"}]},"ts":"1689121904956"} 2023-07-12 00:31:44,959 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-12 00:31:44,962 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 00:31:44,964 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 560 msec 2023-07-12 00:31:45,013 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-12 00:31:45,015 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:45,015 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-12 00:31:45,015 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:45,059 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-12 00:31:45,077 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 00:31:45,082 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 34 msec 2023-07-12 00:31:45,092 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-12 00:31:45,105 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 00:31:45,110 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 17 msec 2023-07-12 00:31:45,124 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 00:31:45,125 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 00:31:45,126 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.526sec 2023-07-12 00:31:45,129 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 00:31:45,131 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 00:31:45,131 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 00:31:45,133 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,40891,1689121902393-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 00:31:45,134 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,40891,1689121902393-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 00:31:45,143 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 00:31:45,157 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ReadOnlyZKClient(139): Connect 0x1d0e3117 to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:31:45,163 DEBUG [Listener at localhost.localdomain/34481] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2f78b06e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:31:45,179 DEBUG [hconnection-0x6e00eb3c-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 00:31:45,190 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:58910, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 00:31:45,200 INFO [Listener at localhost.localdomain/34481] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,40891,1689121902393 2023-07-12 00:31:45,200 INFO [Listener at localhost.localdomain/34481] hbase.TestZooKeeper(121): Starting testRegionServerSessionExpired 2023-07-12 00:31:45,205 INFO [Listener at localhost.localdomain/34481-EventThread] hbase.HBaseTestingUtility$2(2877): Monitor ZKW received event=WatchedEvent state:SyncConnected type:None path:null 2023-07-12 00:31:45,208 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:42559-0x1007e9741810003, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,37983,1689121902477 2023-07-12 00:31:45,208 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:42559-0x1007e9741810003, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:31:45,208 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:31:45,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42559-0x1007e9741810003, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,42559,1689121902528 2023-07-12 00:31:45,209 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,37983,1689121902477] 2023-07-12 00:31:45,209 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase20.apache.org,37983,1689121902477 znode expired, triggering replicatorRemoved event 2023-07-12 00:31:45,209 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,37983,1689121902477; numProcessing=1 2023-07-12 00:31:45,209 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40891] master.ServerManager(357): Server REPORT rejected; currently processing jenkins-hbase20.apache.org,37983,1689121902477 as dead server 2023-07-12 00:31:45,210 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40891] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing jenkins-hbase20.apache.org,37983,1689121902477 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:358) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:247) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:559) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:31:45,210 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40891] ipc.CallRunner(144): callId: 15 service: RegionServerStatusService methodName: RegionServerReport size: 456 connection: 148.251.75.209:49053 deadline: 1689121915208, exception=org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing jenkins-hbase20.apache.org,37983,1689121902477 as dead server 2023-07-12 00:31:45,211 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,37983,1689121902477 already deleted, retry=false 2023-07-12 00:31:45,211 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase20.apache.org,37983,1689121902477 on jenkins-hbase20.apache.org,40891,1689121902393 2023-07-12 00:31:45,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42559-0x1007e9741810003, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,42559,1689121902528 2023-07-12 00:31:45,215 ERROR [RS:0;jenkins-hbase20:37983] helpers.MarkerIgnoringBase(159): ***** ABORTING region server jenkins-hbase20.apache.org,37983,1689121902477: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing jenkins-hbase20.apache.org,37983,1689121902477 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:358) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:247) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:559) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) ***** org.apache.hadoop.hbase.YouAreDeadException: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing jenkins-hbase20.apache.org,37983,1689121902477 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:358) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:247) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:559) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:354) at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1270) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1088) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.YouAreDeadException): org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing jenkins-hbase20.apache.org,37983,1689121902477 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:358) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:247) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:559) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-12 00:31:45,218 ERROR [RS:0;jenkins-hbase20:37983] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 00:31:45,220 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase20.apache.org,37983,1689121902477, splitWal=true, meta=true 2023-07-12 00:31:45,221 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=9 for jenkins-hbase20.apache.org,37983,1689121902477 (carryingMeta=true) jenkins-hbase20.apache.org,37983,1689121902477/CRASHED/regionCount=2/lock=java.util.concurrent.locks.ReentrantReadWriteLock@109c1b74[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-12 00:31:45,222 DEBUG [RS:0;jenkins-hbase20:37983] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-12 00:31:45,225 DEBUG [RS:0;jenkins-hbase20:37983] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-12 00:31:45,226 DEBUG [RS:0;jenkins-hbase20:37983] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-12 00:31:45,226 INFO [PEWorker-2] procedure.ServerCrashProcedure(161): Start pid=9, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase20.apache.org,37983,1689121902477, splitWal=true, meta=true 2023-07-12 00:31:45,226 DEBUG [RS:0;jenkins-hbase20:37983] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-12 00:31:45,227 INFO [RS:0;jenkins-hbase20:37983] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1043857408, "init": 524288000, "max": 2051014656, "used": 450247000 }, "NonHeapMemoryUsage": { "committed": 112549888, "init": 2555904, "max": -1, "used": 110404776 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org" } ] } 2023-07-12 00:31:45,229 INFO [PEWorker-2] procedure.ServerCrashProcedure(300): Splitting WALs pid=9, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase20.apache.org,37983,1689121902477, splitWal=true, meta=true, isMeta: true 2023-07-12 00:31:45,233 DEBUG [PEWorker-2] master.MasterWalManager(318): Renamed region directory: hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting 2023-07-12 00:31:45,235 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase20.apache.org,37983,1689121902477 WAL count=1, meta=true 2023-07-12 00:31:45,236 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=40891] master.MasterRpcServices(612): jenkins-hbase20.apache.org,37983,1689121902477 reported a fatal error: ***** ABORTING region server jenkins-hbase20.apache.org,37983,1689121902477: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing jenkins-hbase20.apache.org,37983,1689121902477 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:358) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:247) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:559) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) ***** Cause: org.apache.hadoop.hbase.YouAreDeadException: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing jenkins-hbase20.apache.org,37983,1689121902477 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:358) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:247) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:559) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:354) at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1270) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1088) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.YouAreDeadException): org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing jenkins-hbase20.apache.org,37983,1689121902477 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:358) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:247) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:559) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-12 00:31:45,239 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta}] 2023-07-12 00:31:45,240 INFO [RS:0;jenkins-hbase20:37983] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,37983,1689121902477' ***** 2023-07-12 00:31:45,240 INFO [RS:0;jenkins-hbase20:37983] regionserver.HRegionServer(2311): STOPPED: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing jenkins-hbase20.apache.org,37983,1689121902477 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:358) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:247) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:559) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:31:45,241 INFO [RS:0;jenkins-hbase20:37983] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 00:31:45,243 INFO [RS:0;jenkins-hbase20:37983] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-12 00:31:45,243 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 00:31:45,243 INFO [RS:0;jenkins-hbase20:37983] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-12 00:31:45,244 INFO [RS:0;jenkins-hbase20:37983] regionserver.HRegionServer(3305): Received CLOSE for 150e8dd2bd05831664bbcb60e0b3f011 2023-07-12 00:31:45,245 INFO [RS:0;jenkins-hbase20:37983] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,37983,1689121902477 2023-07-12 00:31:45,246 DEBUG [RS:0;jenkins-hbase20:37983] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x209b501c to 127.0.0.1:58965 2023-07-12 00:31:45,247 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 150e8dd2bd05831664bbcb60e0b3f011, disabling compactions & flushes 2023-07-12 00:31:45,247 DEBUG [RS:0;jenkins-hbase20:37983] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:31:45,247 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011. 2023-07-12 00:31:45,247 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011. 2023-07-12 00:31:45,247 INFO [RS:0;jenkins-hbase20:37983] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 00:31:45,247 INFO [RS:0;jenkins-hbase20:37983] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 00:31:45,247 INFO [RS:0;jenkins-hbase20:37983] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 00:31:45,247 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011. after waiting 0 ms 2023-07-12 00:31:45,247 INFO [RS:0;jenkins-hbase20:37983] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-12 00:31:45,247 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011. 2023-07-12 00:31:45,248 INFO [RS:0;jenkins-hbase20:37983] regionserver.HRegionServer(1474): Waiting on 2 regions to close 2023-07-12 00:31:45,248 DEBUG [RS:0;jenkins-hbase20:37983] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740, 150e8dd2bd05831664bbcb60e0b3f011=hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011.} 2023-07-12 00:31:45,248 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011. 2023-07-12 00:31:45,249 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 00:31:45,250 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 00:31:45,250 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 00:31:45,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011. 2023-07-12 00:31:45,250 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 00:31:45,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 150e8dd2bd05831664bbcb60e0b3f011: 2023-07-12 00:31:45,250 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 00:31:45,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011. 2023-07-12 00:31:45,250 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1825): Memstore data size is 1292 in region hbase:meta,,1.1588230740 2023-07-12 00:31:45,251 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 00:31:45,251 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 00:31:45,252 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 00:31:45,252 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-12 00:31:45,252 DEBUG [RS:0;jenkins-hbase20:37983] regionserver.HRegionServer(1504): Waiting on 2023-07-12 00:31:45,255 DEBUG [PEWorker-4] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase20.apache.org,42559,1689121902528 2023-07-12 00:31:45,257 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta, worker=jenkins-hbase20.apache.org,42559,1689121902528}] 2023-07-12 00:31:45,305 INFO [Listener at localhost.localdomain/34481-EventThread] hbase.HBaseTestingUtility$2(2877): Monitor ZKW received event=WatchedEvent state:Disconnected type:None path:null 2023-07-12 00:31:45,306 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:37983-0x1007e9741810002, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2023-07-12 00:31:45,306 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:37983-0x1007e9741810002, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2023-07-12 00:31:45,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(630): regionserver:37983-0x1007e9741810002, quorum=127.0.0.1:58965, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2023-07-12 00:31:45,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(630): regionserver:37983-0x1007e9741810002, quorum=127.0.0.1:58965, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2023-07-12 00:31:45,312 INFO [Listener at localhost.localdomain/34481] hbase.HBaseTestingUtility(2892): ZK Closed Session 0x1007e9741810002 2023-07-12 00:31:45,324 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 00:31:45,425 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,42559,1689121902528 2023-07-12 00:31:45,426 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 00:31:45,429 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:58298, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 00:31:45,430 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=42559] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=11 2023-07-12 00:31:45,451 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(300): Splitting hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta, size=0 (0bytes) 2023-07-12 00:31:45,451 WARN [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(451): File hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta might be still open, length is 0 2023-07-12 00:31:45,452 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta 2023-07-12 00:31:45,452 INFO [RS:0;jenkins-hbase20:37983] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,37983,1689121902477; all regions closed. 2023-07-12 00:31:45,456 WARN [IPC Server handler 3 on default port 44295] namenode.FSNamesystem(3291): DIR* NameSystem.internalReleaseLease: File /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta has not been closed. Lease recovery is in progress. RecoveryId = 1013 for block blk_1073741834_1010 2023-07-12 00:31:45,460 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] util.RecoverLeaseFSUtils(175): Failed to recover lease, attempt=0 on file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta after 8ms 2023-07-12 00:31:45,465 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(646): lease for file /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477/jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta is expired, give up org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): Client (=DFSClient_NONMAPREDUCE_154608746_17) is not the lease owner (=DFSClient_NONMAPREDUCE_978775016_17: /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta (inode 16433) [Lease. Holder: DFSClient_NONMAPREDUCE_154608746_17, pending creates: 1] at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2718) at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.completeFileInternal(FSDirWriteFileOp.java:621) at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.completeFile(FSDirWriteFileOp.java:601) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2741) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:935) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:562) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:507) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1034) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:994) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:922) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2833) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1540) at org.apache.hadoop.ipc.Client.call(Client.java:1486) at org.apache.hadoop.ipc.Client.call(Client.java:1385) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy31.complete(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:488) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy34.complete(Unknown Source) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy35.complete(Unknown Source) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy35.complete(Unknown Source) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.completeFile(FanOutOneBlockAsyncDFSOutputHelper.java:637) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:586) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:169) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:746) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:31:45,474 DEBUG [RS:0;jenkins-hbase20:37983] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:31:45,475 INFO [RS:0;jenkins-hbase20:37983] regionserver.LeaseManager(133): Closed leases 2023-07-12 00:31:45,476 INFO [RS:0;jenkins-hbase20:37983] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-12 00:31:45,476 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 00:31:45,477 INFO [RS:0;jenkins-hbase20:37983] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:37983 2023-07-12 00:31:47,068 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:37983-0x1007e9741810002, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Expired, path=null 2023-07-12 00:31:47,068 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:37983-0x1007e9741810002, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Expired, path=null 2023-07-12 00:31:47,069 DEBUG [zk-event-processor-pool-0] regionserver.HRegionServer(2541): Abort already in progress. Ignoring the current request with reason: regionserver:37983-0x1007e9741810002, quorum=127.0.0.1:58965, baseZNode=/hbase regionserver:37983-0x1007e9741810002 received expired from ZooKeeper, aborting 2023-07-12 00:31:47,069 DEBUG [zk-event-processor-pool-0] regionserver.HRegionServer(2541): Abort already in progress. Ignoring the current request with reason: regionserver:37983-0x1007e9741810002, quorum=127.0.0.1:58965, baseZNode=/hbase regionserver:37983-0x1007e9741810002 received expired from ZooKeeper, aborting 2023-07-12 00:31:47,169 WARN [RS:0;jenkins-hbase20:37983] regionserver.HRegionServer(1217): Failed deleting my ephemeral node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/jenkins-hbase20.apache.org,37983,1689121902477 at org.apache.zookeeper.KeeperException.create(KeeperException.java:134) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:1793) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:162) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:902) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:894) at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1694) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1213) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:31:47,170 INFO [RS:0;jenkins-hbase20:37983] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,37983,1689121902477; zookeeper connection closed. 2023-07-12 00:31:47,170 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@582e5cda] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@582e5cda 2023-07-12 00:31:47,222 INFO [Listener at localhost.localdomain/34481-EventThread] hbase.HBaseTestingUtility$2(2877): Monitor ZKW received event=WatchedEvent state:Expired type:None path:null 2023-07-12 00:31:47,370 INFO [Listener at localhost.localdomain/34481] hbase.TestZooKeeper(154): Creating table testRegionServerSessionExpired_1689121907370 2023-07-12 00:31:47,384 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-12 00:31:47,387 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42076, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-12 00:31:47,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40891] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'testRegionServerSessionExpired_1689121907370', {NAME => 'fam', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 00:31:47,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40891] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRegionServerSessionExpired_1689121907370 2023-07-12 00:31:47,402 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRegionServerSessionExpired_1689121907370 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 00:31:47,405 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRegionServerSessionExpired_1689121907370 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 00:31:47,407 WARN [RS-EventLoopGroup-5-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase20.apache.org/148.251.75.209:37983 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:37983 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:31:47,408 DEBUG [RS-EventLoopGroup-5-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase20.apache.org/148.251.75.209:37983 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:37983 2023-07-12 00:31:47,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40891] master.MasterRpcServices(700): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "testRegionServerSessionExpired_1689121907370" procId is: 12 2023-07-12 00:31:47,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40891] master.MasterRpcServices(1230): Checking to see if procedure is done pid=12 2023-07-12 00:31:47,515 DEBUG [PEWorker-1] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase20.apache.org/148.251.75.209:37983 this server is in the failed servers list 2023-07-12 00:31:47,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40891] master.MasterRpcServices(1230): Checking to see if procedure is done pid=12 2023-07-12 00:31:47,721 DEBUG [PEWorker-1] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase20.apache.org/148.251.75.209:37983 this server is in the failed servers list 2023-07-12 00:31:47,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40891] master.MasterRpcServices(1230): Checking to see if procedure is done pid=12 2023-07-12 00:31:48,026 DEBUG [PEWorker-1] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase20.apache.org/148.251.75.209:37983 this server is in the failed servers list 2023-07-12 00:31:48,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40891] master.MasterRpcServices(1230): Checking to see if procedure is done pid=12 2023-07-12 00:31:48,531 DEBUG [PEWorker-1] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase20.apache.org/148.251.75.209:37983 this server is in the failed servers list 2023-07-12 00:31:48,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40891] master.MasterRpcServices(1230): Checking to see if procedure is done pid=12 2023-07-12 00:31:49,461 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=1 on file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta after 4009ms 2023-07-12 00:31:49,472 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(311): Open hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta took 4022ms 2023-07-12 00:31:49,491 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-12 00:31:49,495 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta so closing down 2023-07-12 00:31:49,495 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-12 00:31:49,502 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0-Writer-2] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-12 00:31:49,505 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta.temp 2023-07-12 00:31:49,507 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-12 00:31:49,523 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta.temp (wrote 5 edits, skipped 0 edits in 0 ms) 2023-07-12 00:31:49,525 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta.temp to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/0000000000000000008 2023-07-12 00:31:49,526 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(423): Processed 6 edits across 1 Regions in 53 ms; skipped=1; WAL=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta, size=0, length=0, corrupted=false, cancelled=false 2023-07-12 00:31:49,526 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta, journal: Splitting hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta, size=0 (0bytes) at 1689121905451Finishing writing output for hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta so closing down at 1689121909495 (+4044 ms)Creating recovered edits writer path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta.temp at 1689121909505 (+10 ms)3 split writer threads finished at 1689121909507 (+2 ms)Closed recovered edits writer path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta.temp (wrote 5 edits, skipped 0 edits in 0 ms) at 1689121909523 (+16 ms)Rename recovered edits hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta.temp to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/0000000000000000008 at 1689121909525 (+2 ms)Processed 6 edits across 1 Regions in 53 ms; skipped=1; WAL=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta, size=0, length=0, corrupted=false, cancelled=false at 1689121909526 (+1 ms) 2023-07-12 00:31:49,527 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta 2023-07-12 00:31:49,529 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=11 2023-07-12 00:31:49,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40891] master.HMaster(3641): Remote procedure done, pid=11 2023-07-12 00:31:49,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40891] master.MasterRpcServices(1230): Checking to see if procedure is done pid=12 2023-07-12 00:31:49,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40891] wal.WALSplitUtil(141): Moved hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/oldWALs 2023-07-12 00:31:49,540 WARN [RS-EventLoopGroup-5-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase20.apache.org/148.251.75.209:37983 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:37983 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:31:49,541 DEBUG [RS-EventLoopGroup-5-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase20.apache.org/148.251.75.209:37983 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:37983 2023-07-12 00:31:49,544 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-12 00:31:49,544 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta, worker=jenkins-hbase20.apache.org,42559,1689121902528 in 4.2840 sec 2023-07-12 00:31:49,546 DEBUG [PEWorker-2] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase20.apache.org,42559,1689121902528 2023-07-12 00:31:49,551 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-12 00:31:49,551 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; SplitWALProcedure jenkins-hbase20.apache.org%2C37983%2C1689121902477.meta.1689121904154.meta, worker=jenkins-hbase20.apache.org,42559,1689121902528 in 4.3100 sec 2023-07-12 00:31:49,554 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting dir is empty, no logs to split. 2023-07-12 00:31:49,554 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase20.apache.org,37983,1689121902477 WAL count=0, meta=true 2023-07-12 00:31:49,554 DEBUG [PEWorker-4] procedure.ServerCrashProcedure(290): Check if jenkins-hbase20.apache.org,37983,1689121902477 WAL splitting is done? wals=0, meta=true 2023-07-12 00:31:49,556 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-12 00:31:49,557 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-12 00:31:49,559 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-12 00:31:49,709 DEBUG [jenkins-hbase20:40891] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-12 00:31:49,709 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,42559,1689121902528, state=OPENING 2023-07-12 00:31:49,712 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 00:31:49,712 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 00:31:49,712 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,42559,1689121902528}] 2023-07-12 00:31:49,712 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 00:31:49,712 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 00:31:49,725 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 00:31:49,878 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-12 00:31:49,878 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:31:49,883 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C42559%2C1689121902528.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,42559,1689121902528, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/oldWALs, maxLogs=32 2023-07-12 00:31:49,902 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:31:49,902 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:31:49,906 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,42559,1689121902528/jenkins-hbase20.apache.org%2C42559%2C1689121902528.meta.1689121909885.meta 2023-07-12 00:31:49,906 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK], DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK]] 2023-07-12 00:31:49,907 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-12 00:31:49,907 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-12 00:31:49,907 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-12 00:31:49,908 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-12 00:31:49,908 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-12 00:31:49,908 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:31:49,908 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-12 00:31:49,908 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-12 00:31:49,910 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 00:31:49,913 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/info 2023-07-12 00:31:49,913 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/info 2023-07-12 00:31:49,914 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 00:31:49,915 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:31:49,915 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 00:31:49,917 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/rep_barrier 2023-07-12 00:31:49,917 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/rep_barrier 2023-07-12 00:31:49,918 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 00:31:49,918 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:31:49,918 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 00:31:49,921 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/table 2023-07-12 00:31:49,921 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/table 2023-07-12 00:31:49,921 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 00:31:49,923 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:31:49,924 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740 2023-07-12 00:31:49,928 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740 2023-07-12 00:31:49,930 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/0000000000000000008 2023-07-12 00:31:49,937 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5464): Applied 12, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=8, path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/0000000000000000008 2023-07-12 00:31:49,938 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.26 KB heapSize=2.89 KB 2023-07-12 00:31:49,988 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.17 KB at sequenceid=8 (bloomFilter=false), to=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/.tmp/info/7bbe08405ef54abb8da07bed6abbe741 2023-07-12 00:31:50,049 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=94 B at sequenceid=8 (bloomFilter=false), to=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/.tmp/table/01b8cd7d101c448b894a9c31de55e7f7 2023-07-12 00:31:50,059 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/.tmp/info/7bbe08405ef54abb8da07bed6abbe741 as hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/info/7bbe08405ef54abb8da07bed6abbe741 2023-07-12 00:31:50,069 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/info/7bbe08405ef54abb8da07bed6abbe741, entries=10, sequenceid=8, filesize=5.9 K 2023-07-12 00:31:50,072 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/.tmp/table/01b8cd7d101c448b894a9c31de55e7f7 as hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/table/01b8cd7d101c448b894a9c31de55e7f7 2023-07-12 00:31:50,082 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/table/01b8cd7d101c448b894a9c31de55e7f7, entries=2, sequenceid=8, filesize=4.7 K 2023-07-12 00:31:50,082 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.26 KB/1292, heapSize ~2.61 KB/2672, currentSize=0 B/0 for 1588230740 in 144ms, sequenceid=8, compaction requested=false; wal=null 2023-07-12 00:31:50,084 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-12 00:31:50,087 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/0000000000000000008 2023-07-12 00:31:50,090 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 00:31:50,092 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 00:31:50,097 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-12 00:31:50,098 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=9; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73780432, jitterRate=0.09941411018371582}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 00:31:50,099 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 00:31:50,100 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=14, masterSystemTime=1689121909868 2023-07-12 00:31:50,103 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-12 00:31:50,103 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-12 00:31:50,104 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,42559,1689121902528, state=OPEN 2023-07-12 00:31:50,105 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 00:31:50,105 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 00:31:50,105 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 00:31:50,105 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 00:31:50,109 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-12 00:31:50,109 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,42559,1689121902528 in 393 msec 2023-07-12 00:31:50,113 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=9 2023-07-12 00:31:50,113 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 553 msec 2023-07-12 00:31:50,113 INFO [PEWorker-2] procedure.ServerCrashProcedure(199): jenkins-hbase20.apache.org,37983,1689121902477 had 2 regions 2023-07-12 00:31:50,115 INFO [PEWorker-2] procedure.ServerCrashProcedure(300): Splitting WALs pid=9, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase20.apache.org,37983,1689121902477, splitWal=true, meta=true, isMeta: false 2023-07-12 00:31:50,117 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase20.apache.org,37983,1689121902477 WAL count=1, meta=false 2023-07-12 00:31:50,117 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=9, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase20.apache.org%2C37983%2C1689121902477.1689121903986}] 2023-07-12 00:31:50,120 DEBUG [PEWorker-2] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase20.apache.org,42559,1689121902528 2023-07-12 00:31:50,121 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase20.apache.org%2C37983%2C1689121902477.1689121903986, worker=jenkins-hbase20.apache.org,42559,1689121902528}] 2023-07-12 00:31:50,274 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=42559] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=16 2023-07-12 00:31:50,294 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(300): Splitting hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.1689121903986, size=609 (609bytes) 2023-07-12 00:31:50,294 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.1689121903986 2023-07-12 00:31:50,295 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.1689121903986 after 1ms 2023-07-12 00:31:50,299 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(311): Open hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.1689121903986 took 5ms 2023-07-12 00:31:50,303 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(346): Last flushed sequenceid for 150e8dd2bd05831664bbcb60e0b3f011: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-12 00:31:50,304 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.1689121903986 so closing down 2023-07-12 00:31:50,304 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-12 00:31:50,309 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C37983%2C1689121902477.1689121903986.temp 2023-07-12 00:31:50,309 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-12 00:31:50,325 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C37983%2C1689121902477.1689121903986.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-12 00:31:50,327 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C37983%2C1689121902477.1689121903986.temp to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011/recovered.edits/0000000000000000005 2023-07-12 00:31:50,327 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(423): Processed 3 edits across 1 Regions in 28 ms; skipped=1; WAL=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.1689121903986, size=609, length=609, corrupted=false, cancelled=false 2023-07-12 00:31:50,328 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.1689121903986, journal: Splitting hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.1689121903986, size=609 (609bytes) at 1689121910294Finishing writing output for hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.1689121903986 so closing down at 1689121910304 (+10 ms)Creating recovered edits writer path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C37983%2C1689121902477.1689121903986.temp at 1689121910309 (+5 ms)3 split writer threads finished at 1689121910310 (+1 ms)Closed recovered edits writer path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C37983%2C1689121902477.1689121903986.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689121910325 (+15 ms)Rename recovered edits hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C37983%2C1689121902477.1689121903986.temp to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011/recovered.edits/0000000000000000005 at 1689121910327 (+2 ms)Processed 3 edits across 1 Regions in 28 ms; skipped=1; WAL=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.1689121903986, size=609, length=609, corrupted=false, cancelled=false at 1689121910328 (+1 ms) 2023-07-12 00:31:50,328 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.1689121903986 2023-07-12 00:31:50,328 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=16 2023-07-12 00:31:50,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40891] master.HMaster(3641): Remote procedure done, pid=16 2023-07-12 00:31:50,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40891] wal.WALSplitUtil(141): Moved hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting/jenkins-hbase20.apache.org%2C37983%2C1689121902477.1689121903986 to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/oldWALs 2023-07-12 00:31:50,338 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-07-12 00:31:50,339 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase20.apache.org%2C37983%2C1689121902477.1689121903986, worker=jenkins-hbase20.apache.org,42559,1689121902528 in 214 msec 2023-07-12 00:31:50,340 DEBUG [PEWorker-5] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase20.apache.org,42559,1689121902528 2023-07-12 00:31:50,344 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=9 2023-07-12 00:31:50,344 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=9, state=SUCCESS; SplitWALProcedure jenkins-hbase20.apache.org%2C37983%2C1689121902477.1689121903986, worker=jenkins-hbase20.apache.org,42559,1689121902528 in 224 msec 2023-07-12 00:31:50,346 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37983,1689121902477-splitting dir is empty, no logs to split. 2023-07-12 00:31:50,346 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase20.apache.org,37983,1689121902477 WAL count=0, meta=false 2023-07-12 00:31:50,346 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase20.apache.org,37983,1689121902477 WAL splitting is done? wals=0, meta=false 2023-07-12 00:31:50,349 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=150e8dd2bd05831664bbcb60e0b3f011, ASSIGN}] 2023-07-12 00:31:50,350 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=150e8dd2bd05831664bbcb60e0b3f011, ASSIGN 2023-07-12 00:31:50,351 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=150e8dd2bd05831664bbcb60e0b3f011, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-12 00:31:50,501 DEBUG [jenkins-hbase20:40891] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-12 00:31:50,502 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=150e8dd2bd05831664bbcb60e0b3f011, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42559,1689121902528 2023-07-12 00:31:50,503 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689121910502"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121910502"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121910502"}]},"ts":"1689121910502"} 2023-07-12 00:31:50,508 DEBUG [PEWorker-3] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 00:31:50,515 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52558, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 00:31:50,520 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 150e8dd2bd05831664bbcb60e0b3f011, server=jenkins-hbase20.apache.org,42559,1689121902528}] 2023-07-12 00:31:50,683 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011. 2023-07-12 00:31:50,684 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 150e8dd2bd05831664bbcb60e0b3f011, NAME => 'hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011.', STARTKEY => '', ENDKEY => ''} 2023-07-12 00:31:50,684 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 150e8dd2bd05831664bbcb60e0b3f011 2023-07-12 00:31:50,684 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:31:50,684 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 150e8dd2bd05831664bbcb60e0b3f011 2023-07-12 00:31:50,685 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 150e8dd2bd05831664bbcb60e0b3f011 2023-07-12 00:31:50,687 INFO [StoreOpener-150e8dd2bd05831664bbcb60e0b3f011-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 150e8dd2bd05831664bbcb60e0b3f011 2023-07-12 00:31:50,688 DEBUG [StoreOpener-150e8dd2bd05831664bbcb60e0b3f011-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011/info 2023-07-12 00:31:50,688 DEBUG [StoreOpener-150e8dd2bd05831664bbcb60e0b3f011-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011/info 2023-07-12 00:31:50,689 INFO [StoreOpener-150e8dd2bd05831664bbcb60e0b3f011-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 150e8dd2bd05831664bbcb60e0b3f011 columnFamilyName info 2023-07-12 00:31:50,689 INFO [StoreOpener-150e8dd2bd05831664bbcb60e0b3f011-1] regionserver.HStore(310): Store=150e8dd2bd05831664bbcb60e0b3f011/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:31:50,692 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011 2023-07-12 00:31:50,696 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011 2023-07-12 00:31:50,697 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011/recovered.edits/0000000000000000005 2023-07-12 00:31:50,701 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5464): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011/recovered.edits/0000000000000000005 2023-07-12 00:31:50,701 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 150e8dd2bd05831664bbcb60e0b3f011 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-12 00:31:50,724 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011/.tmp/info/7406a11381a54c7aaeb089da78863d05 2023-07-12 00:31:50,736 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011/.tmp/info/7406a11381a54c7aaeb089da78863d05 as hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011/info/7406a11381a54c7aaeb089da78863d05 2023-07-12 00:31:50,745 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011/info/7406a11381a54c7aaeb089da78863d05, entries=2, sequenceid=5, filesize=4.8 K 2023-07-12 00:31:50,746 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 150e8dd2bd05831664bbcb60e0b3f011 in 45ms, sequenceid=5, compaction requested=false; wal=null 2023-07-12 00:31:50,746 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-12 00:31:50,748 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011/recovered.edits/0000000000000000005 2023-07-12 00:31:50,753 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 150e8dd2bd05831664bbcb60e0b3f011 2023-07-12 00:31:50,756 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/150e8dd2bd05831664bbcb60e0b3f011/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2023-07-12 00:31:50,758 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 150e8dd2bd05831664bbcb60e0b3f011; next sequenceid=6; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72823614, jitterRate=0.0851564109325409}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:31:50,758 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 150e8dd2bd05831664bbcb60e0b3f011: 2023-07-12 00:31:50,759 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011., pid=18, masterSystemTime=1689121910674 2023-07-12 00:31:50,761 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011. 2023-07-12 00:31:50,762 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011. 2023-07-12 00:31:50,762 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=150e8dd2bd05831664bbcb60e0b3f011, regionState=OPEN, openSeqNum=6, regionLocation=jenkins-hbase20.apache.org,42559,1689121902528 2023-07-12 00:31:50,763 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689121910762"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121910762"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121910762"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121910762"}]},"ts":"1689121910762"} 2023-07-12 00:31:50,769 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-07-12 00:31:50,769 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 150e8dd2bd05831664bbcb60e0b3f011, server=jenkins-hbase20.apache.org,42559,1689121902528 in 246 msec 2023-07-12 00:31:50,772 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=9 2023-07-12 00:31:50,773 INFO [PEWorker-3] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase20.apache.org,37983,1689121902477 after splitting done 2023-07-12 00:31:50,773 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=150e8dd2bd05831664bbcb60e0b3f011, ASSIGN in 420 msec 2023-07-12 00:31:50,773 DEBUG [PEWorker-3] master.DeadServer(114): Removed jenkins-hbase20.apache.org,37983,1689121902477 from processing; numProcessing=0 2023-07-12 00:31:50,775 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; ServerCrashProcedure jenkins-hbase20.apache.org,37983,1689121902477, splitWal=true, meta=true in 5.5600 sec 2023-07-12 00:31:51,340 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 00:31:51,341 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-12 00:31:51,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40891] master.MasterRpcServices(1230): Checking to see if procedure is done pid=12 2023-07-12 00:31:51,549 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionServerSessionExpired_1689121907370/cc777a4cbddcbb1865eb503ec47f25b1 2023-07-12 00:31:51,550 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionServerSessionExpired_1689121907370/cc777a4cbddcbb1865eb503ec47f25b1 empty. 2023-07-12 00:31:51,552 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionServerSessionExpired_1689121907370/cc777a4cbddcbb1865eb503ec47f25b1 2023-07-12 00:31:51,552 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRegionServerSessionExpired_1689121907370 regions 2023-07-12 00:31:51,573 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionServerSessionExpired_1689121907370/.tabledesc/.tableinfo.0000000001 2023-07-12 00:31:51,574 INFO [RegionOpenAndInit-testRegionServerSessionExpired_1689121907370-pool-0] regionserver.HRegion(7675): creating {ENCODED => cc777a4cbddcbb1865eb503ec47f25b1, NAME => 'testRegionServerSessionExpired_1689121907370,,1689121907394.cc777a4cbddcbb1865eb503ec47f25b1.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testRegionServerSessionExpired_1689121907370', {NAME => 'fam', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp 2023-07-12 00:31:51,592 DEBUG [RegionOpenAndInit-testRegionServerSessionExpired_1689121907370-pool-0] regionserver.HRegion(866): Instantiated testRegionServerSessionExpired_1689121907370,,1689121907394.cc777a4cbddcbb1865eb503ec47f25b1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:31:51,592 DEBUG [RegionOpenAndInit-testRegionServerSessionExpired_1689121907370-pool-0] regionserver.HRegion(1604): Closing cc777a4cbddcbb1865eb503ec47f25b1, disabling compactions & flushes 2023-07-12 00:31:51,592 INFO [RegionOpenAndInit-testRegionServerSessionExpired_1689121907370-pool-0] regionserver.HRegion(1626): Closing region testRegionServerSessionExpired_1689121907370,,1689121907394.cc777a4cbddcbb1865eb503ec47f25b1. 2023-07-12 00:31:51,592 DEBUG [RegionOpenAndInit-testRegionServerSessionExpired_1689121907370-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRegionServerSessionExpired_1689121907370,,1689121907394.cc777a4cbddcbb1865eb503ec47f25b1. 2023-07-12 00:31:51,592 DEBUG [RegionOpenAndInit-testRegionServerSessionExpired_1689121907370-pool-0] regionserver.HRegion(1714): Acquired close lock on testRegionServerSessionExpired_1689121907370,,1689121907394.cc777a4cbddcbb1865eb503ec47f25b1. after waiting 0 ms 2023-07-12 00:31:51,592 DEBUG [RegionOpenAndInit-testRegionServerSessionExpired_1689121907370-pool-0] regionserver.HRegion(1724): Updates disabled for region testRegionServerSessionExpired_1689121907370,,1689121907394.cc777a4cbddcbb1865eb503ec47f25b1. 2023-07-12 00:31:51,592 INFO [RegionOpenAndInit-testRegionServerSessionExpired_1689121907370-pool-0] regionserver.HRegion(1838): Closed testRegionServerSessionExpired_1689121907370,,1689121907394.cc777a4cbddcbb1865eb503ec47f25b1. 2023-07-12 00:31:51,592 DEBUG [RegionOpenAndInit-testRegionServerSessionExpired_1689121907370-pool-0] regionserver.HRegion(1558): Region close journal for cc777a4cbddcbb1865eb503ec47f25b1: 2023-07-12 00:31:51,600 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRegionServerSessionExpired_1689121907370 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 00:31:51,602 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRegionServerSessionExpired_1689121907370,,1689121907394.cc777a4cbddcbb1865eb503ec47f25b1.","families":{"info":[{"qualifier":"regioninfo","vlen":78,"tag":[],"timestamp":"1689121911602"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689121911602"}]},"ts":"1689121911602"} 2023-07-12 00:31:51,605 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 00:31:51,606 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRegionServerSessionExpired_1689121907370 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 00:31:51,607 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRegionServerSessionExpired_1689121907370","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689121911606"}]},"ts":"1689121911606"} 2023-07-12 00:31:51,608 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRegionServerSessionExpired_1689121907370, state=ENABLING in hbase:meta 2023-07-12 00:31:51,652 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRegionServerSessionExpired_1689121907370, region=cc777a4cbddcbb1865eb503ec47f25b1, ASSIGN}] 2023-07-12 00:31:51,657 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRegionServerSessionExpired_1689121907370, region=cc777a4cbddcbb1865eb503ec47f25b1, ASSIGN 2023-07-12 00:31:51,659 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRegionServerSessionExpired_1689121907370, region=cc777a4cbddcbb1865eb503ec47f25b1, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,42559,1689121902528; forceNewPlan=false, retain=false 2023-07-12 00:31:51,811 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=cc777a4cbddcbb1865eb503ec47f25b1, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42559,1689121902528 2023-07-12 00:31:51,812 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRegionServerSessionExpired_1689121907370,,1689121907394.cc777a4cbddcbb1865eb503ec47f25b1.","families":{"info":[{"qualifier":"regioninfo","vlen":78,"tag":[],"timestamp":"1689121911811"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121911811"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121911811"}]},"ts":"1689121911811"} 2023-07-12 00:31:51,816 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE; OpenRegionProcedure cc777a4cbddcbb1865eb503ec47f25b1, server=jenkins-hbase20.apache.org,42559,1689121902528}] 2023-07-12 00:31:51,984 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open testRegionServerSessionExpired_1689121907370,,1689121907394.cc777a4cbddcbb1865eb503ec47f25b1. 2023-07-12 00:31:51,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => cc777a4cbddcbb1865eb503ec47f25b1, NAME => 'testRegionServerSessionExpired_1689121907370,,1689121907394.cc777a4cbddcbb1865eb503ec47f25b1.', STARTKEY => '', ENDKEY => ''} 2023-07-12 00:31:51,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRegionServerSessionExpired_1689121907370 cc777a4cbddcbb1865eb503ec47f25b1 2023-07-12 00:31:51,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated testRegionServerSessionExpired_1689121907370,,1689121907394.cc777a4cbddcbb1865eb503ec47f25b1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:31:51,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for cc777a4cbddcbb1865eb503ec47f25b1 2023-07-12 00:31:51,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for cc777a4cbddcbb1865eb503ec47f25b1 2023-07-12 00:31:51,988 INFO [StoreOpener-cc777a4cbddcbb1865eb503ec47f25b1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family fam of region cc777a4cbddcbb1865eb503ec47f25b1 2023-07-12 00:31:51,991 DEBUG [StoreOpener-cc777a4cbddcbb1865eb503ec47f25b1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionServerSessionExpired_1689121907370/cc777a4cbddcbb1865eb503ec47f25b1/fam 2023-07-12 00:31:51,991 DEBUG [StoreOpener-cc777a4cbddcbb1865eb503ec47f25b1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionServerSessionExpired_1689121907370/cc777a4cbddcbb1865eb503ec47f25b1/fam 2023-07-12 00:31:51,992 INFO [StoreOpener-cc777a4cbddcbb1865eb503ec47f25b1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cc777a4cbddcbb1865eb503ec47f25b1 columnFamilyName fam 2023-07-12 00:31:51,993 INFO [StoreOpener-cc777a4cbddcbb1865eb503ec47f25b1-1] regionserver.HStore(310): Store=cc777a4cbddcbb1865eb503ec47f25b1/fam, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:31:51,996 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionServerSessionExpired_1689121907370/cc777a4cbddcbb1865eb503ec47f25b1 2023-07-12 00:31:51,998 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionServerSessionExpired_1689121907370/cc777a4cbddcbb1865eb503ec47f25b1 2023-07-12 00:31:52,003 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for cc777a4cbddcbb1865eb503ec47f25b1 2023-07-12 00:31:52,006 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionServerSessionExpired_1689121907370/cc777a4cbddcbb1865eb503ec47f25b1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:31:52,007 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened cc777a4cbddcbb1865eb503ec47f25b1; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58845258, jitterRate=-0.12313732504844666}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:31:52,007 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for cc777a4cbddcbb1865eb503ec47f25b1: 2023-07-12 00:31:52,009 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRegionServerSessionExpired_1689121907370,,1689121907394.cc777a4cbddcbb1865eb503ec47f25b1., pid=20, masterSystemTime=1689121911971 2023-07-12 00:31:52,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRegionServerSessionExpired_1689121907370,,1689121907394.cc777a4cbddcbb1865eb503ec47f25b1. 2023-07-12 00:31:52,012 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened testRegionServerSessionExpired_1689121907370,,1689121907394.cc777a4cbddcbb1865eb503ec47f25b1. 2023-07-12 00:31:52,013 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=cc777a4cbddcbb1865eb503ec47f25b1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,42559,1689121902528 2023-07-12 00:31:52,014 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRegionServerSessionExpired_1689121907370,,1689121907394.cc777a4cbddcbb1865eb503ec47f25b1.","families":{"info":[{"qualifier":"regioninfo","vlen":78,"tag":[],"timestamp":"1689121912013"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121912013"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121912013"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121912013"}]},"ts":"1689121912013"} 2023-07-12 00:31:52,019 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-07-12 00:31:52,019 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; OpenRegionProcedure cc777a4cbddcbb1865eb503ec47f25b1, server=jenkins-hbase20.apache.org,42559,1689121902528 in 200 msec 2023-07-12 00:31:52,022 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=12 2023-07-12 00:31:52,023 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=testRegionServerSessionExpired_1689121907370, region=cc777a4cbddcbb1865eb503ec47f25b1, ASSIGN in 367 msec 2023-07-12 00:31:52,024 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRegionServerSessionExpired_1689121907370 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 00:31:52,025 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRegionServerSessionExpired_1689121907370","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689121912025"}]},"ts":"1689121912025"} 2023-07-12 00:31:52,027 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRegionServerSessionExpired_1689121907370, state=ENABLED in hbase:meta 2023-07-12 00:31:52,030 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRegionServerSessionExpired_1689121907370 execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 00:31:52,032 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; CreateTableProcedure table=testRegionServerSessionExpired_1689121907370 in 4.6320 sec 2023-07-12 00:31:55,419 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 00:31:55,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40891] master.MasterRpcServices(1230): Checking to see if procedure is done pid=12 2023-07-12 00:31:55,541 INFO [Listener at localhost.localdomain/34481] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testRegionServerSessionExpired_1689121907370, procId: 12 completed 2023-07-12 00:31:55,541 INFO [Listener at localhost.localdomain/34481] hbase.TestZooKeeper(165): Putting table testRegionServerSessionExpired_1689121907370 2023-07-12 00:31:55,543 WARN [RPCClient-NioEventLoopGroup-6-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase20.apache.org/148.251.75.209:37983 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:37983 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:31:55,544 DEBUG [RPCClient-NioEventLoopGroup-6-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase20.apache.org/148.251.75.209:37983 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:37983 2023-07-12 00:31:55,648 DEBUG [hconnection-0x6e00eb3c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 00:31:55,655 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52560, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 00:31:55,663 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): ***** ABORTING master jenkins-hbase20.apache.org,35377,1689121901202: killAll ***** 2023-07-12 00:31:55,663 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,35377,1689121901202' ***** 2023-07-12 00:31:55,663 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2311): STOPPED: Stopped by Listener at localhost.localdomain/34481 2023-07-12 00:31:55,664 DEBUG [M:0;jenkins-hbase20:35377] zookeeper.ReadOnlyZKClient(139): Connect 0x5126c7d6 to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:31:55,664 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 00:31:55,664 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): ***** ABORTING master jenkins-hbase20.apache.org,40891,1689121902393: killAll ***** 2023-07-12 00:31:55,664 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,40891,1689121902393' ***** 2023-07-12 00:31:55,664 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2311): STOPPED: Stopped by Listener at localhost.localdomain/34481 2023-07-12 00:31:55,666 DEBUG [M:1;jenkins-hbase20:40891] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7de2c1db, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-12 00:31:55,667 INFO [M:1;jenkins-hbase20:40891] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,40891,1689121902393 2023-07-12 00:31:55,667 DEBUG [M:1;jenkins-hbase20:40891] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x056bd26c to 127.0.0.1:58965 2023-07-12 00:31:55,667 DEBUG [M:1;jenkins-hbase20:40891] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:31:55,667 INFO [M:1;jenkins-hbase20:40891] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,40891,1689121902393; all regions closed. 2023-07-12 00:31:55,667 DEBUG [M:1;jenkins-hbase20:40891] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:31:55,668 DEBUG [M:1;jenkins-hbase20:40891] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 00:31:55,668 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 00:31:55,668 DEBUG [M:1;jenkins-hbase20:40891] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 00:31:55,668 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689121903637] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689121903637,5,FailOnTimeoutGroup] 2023-07-12 00:31:55,668 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689121903635] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689121903635,5,FailOnTimeoutGroup] 2023-07-12 00:31:55,668 INFO [M:1;jenkins-hbase20:40891] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 00:31:55,670 INFO [M:1;jenkins-hbase20:40891] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 00:31:55,670 INFO [M:1;jenkins-hbase20:40891] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-07-12 00:31:55,670 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:31:55,670 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:31:55,670 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:42559-0x1007e9741810003, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:31:55,670 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:55,670 DEBUG [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2541): Abort already in progress. Ignoring the current request with reason: killAll 2023-07-12 00:31:55,670 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase20.apache.org,42559,1689121902528: killAll ***** 2023-07-12 00:31:55,671 DEBUG [M:1;jenkins-hbase20:40891] master.HMaster(1512): Stopping service threads 2023-07-12 00:31:55,671 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 00:31:55,671 INFO [M:1;jenkins-hbase20:40891] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 00:31:55,671 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-12 00:31:55,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:31:55,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:31:55,672 INFO [M:1;jenkins-hbase20:40891] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 00:31:55,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42559-0x1007e9741810003, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:31:55,671 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:55,673 DEBUG [M:0;jenkins-hbase20:35377] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2e8783b4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:31:55,673 DEBUG [M:0;jenkins-hbase20:35377] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@60bd572c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-12 00:31:55,673 INFO [M:0;jenkins-hbase20:35377] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,35377,1689121901202 2023-07-12 00:31:55,674 DEBUG [M:0;jenkins-hbase20:35377] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5126c7d6 to 127.0.0.1:58965 2023-07-12 00:31:55,674 DEBUG [M:0;jenkins-hbase20:35377] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:31:55,674 INFO [M:0;jenkins-hbase20:35377] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,35377,1689121901202; all regions closed. 2023-07-12 00:31:55,674 DEBUG [M:0;jenkins-hbase20:35377] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:31:55,674 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-12 00:31:55,674 INFO [M:0;jenkins-hbase20:35377] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-07-12 00:31:55,674 DEBUG [M:0;jenkins-hbase20:35377] master.HMaster(1512): Stopping service threads 2023-07-12 00:31:55,676 ERROR [M:1;jenkins-hbase20:40891] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] 2023-07-12 00:31:55,676 INFO [M:1;jenkins-hbase20:40891] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=true 2023-07-12 00:31:55,677 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 00:31:55,678 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-12 00:31:55,678 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-12 00:31:55,680 DEBUG [M:1;jenkins-hbase20:40891] zookeeper.ZKUtil(398): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 00:31:55,681 DEBUG [M:0;jenkins-hbase20:35377] zookeeper.ZKUtil(398): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 00:31:55,681 WARN [M:1;jenkins-hbase20:40891] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 00:31:55,681 WARN [M:0;jenkins-hbase20:35377] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 00:31:55,681 INFO [M:1;jenkins-hbase20:40891] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 00:31:55,681 INFO [M:0;jenkins-hbase20:35377] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:35377 2023-07-12 00:31:55,682 INFO [M:1;jenkins-hbase20:40891] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=true 2023-07-12 00:31:55,688 DEBUG [M:1;jenkins-hbase20:40891] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 00:31:55,688 INFO [M:1;jenkins-hbase20:40891] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:31:55,688 DEBUG [M:1;jenkins-hbase20:40891] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:31:55,688 DEBUG [M:1;jenkins-hbase20:40891] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 00:31:55,688 DEBUG [M:1;jenkins-hbase20:40891] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:31:55,688 ERROR [M:1;jenkins-hbase20:40891] regionserver.HRegion(1825): Memstore data size is 72260 in region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:31:55,689 INFO [M:1;jenkins-hbase20:40891] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:31:55,689 DEBUG [M:1;jenkins-hbase20:40891] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 00:31:55,689 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 00:31:55,689 INFO [M:1;jenkins-hbase20:40891] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 00:31:55,690 INFO [M:1;jenkins-hbase20:40891] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:40891 2023-07-12 00:31:55,692 DEBUG [M:0;jenkins-hbase20:35377] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,35377,1689121901202 already deleted, retry=false 2023-07-12 00:31:55,695 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1047003136, "init": 524288000, "max": 2051014656, "used": 223829864 }, "NonHeapMemoryUsage": { "committed": 122511360, "init": 2555904, "max": -1, "used": 120225408 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 0, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 7908, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 0, "ProcessCallTime_num_ops": 33, "ProcessCallTime_min": 1, "ProcessCallTime_max": 11, "ProcessCallTime_mean": 3, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 2, "ProcessCallTime_75th_percentile": 7, "ProcessCallTime_90th_percentile": 8, "ProcessCallTime_95th_percentile": 11, "ProcessCallTime_98th_percentile": 11, "ProcessCallTime_99th_percentile": 11, "ProcessCallTime_99.9th_percentile": 11, "ProcessCallTime_TimeRangeCount_0-1": 4, "ProcessCallTime_TimeRangeCount_1-3": 3, "ProcessCallTime_TimeRangeCount_3-10": 4, "ProcessCallTime_TimeRangeCount_10-30": 1, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 33, "QueueCallTime_min": 0, "QueueCallTime_max": 0, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 0, "QueueCallTime_99th_percentile": 0, "QueueCallTime_99.9th_percentile": 0, "QueueCallTime_TimeRangeCount_0-1": 12, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 33, "TotalCallTime_min": 1, "TotalCallTime_max": 11, "TotalCallTime_mean": 3, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 2, "TotalCallTime_75th_percentile": 7, "TotalCallTime_90th_percentile": 8, "TotalCallTime_95th_percentile": 11, "TotalCallTime_98th_percentile": 11, "TotalCallTime_99th_percentile": 11, "TotalCallTime_99.9th_percentile": 11, "TotalCallTime_TimeRangeCount_0-1": 4, "TotalCallTime_TimeRangeCount_1-3": 3, "TotalCallTime_TimeRangeCount_3-10": 4, "TotalCallTime_TimeRangeCount_10-30": 1, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 33, "ResponseSize_min": 0, "ResponseSize_max": 4, "ResponseSize_mean": 1, "ResponseSize_25th_percentile": 0, "ResponseSize_median": 1, "ResponseSize_75th_percentile": 2, "ResponseSize_90th_percentile": 2, "ResponseSize_95th_percentile": 3, "ResponseSize_98th_percentile": 3, "ResponseSize_99th_percentile": 3, "ResponseSize_99.9th_percentile": 3, "ResponseSize_SizeRangeCount_0-10": 12, "exceptions.UnknownScannerException": 0, "exceptions": 0, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 0, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 33, "RequestSize_min": 80, "RequestSize_max": 406, "RequestSize_mean": 224, "RequestSize_25th_percentile": 127, "RequestSize_median": 161, "RequestSize_75th_percentile": 319, "RequestSize_90th_percentile": 379, "RequestSize_95th_percentile": 392, "RequestSize_98th_percentile": 400, "RequestSize_99th_percentile": 403, "RequestSize_99.9th_percentile": 405, "RequestSize_SizeRangeCount_10-100": 2, "RequestSize_SizeRangeCount_100-1000": 10, "sentBytes": 2109 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:58965", "tag.serverName": "jenkins-hbase20.apache.org,42559,1689121902528", "tag.clusterId": "b94c042f-d510-47b7-9d4c-011dde64d627", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "regionCount": 3, "storeCount": 5, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 3, "maxStoreFileCount": 1, "memStoreSize": 2711, "storeFileSize": 15699, "maxStoreFileAge": 3803, "minStoreFileAge": 3067, "avgStoreFileAge": 3537, "numReferenceFiles": 0, "regionServerStartTime": 1689121902528, "averageRegionSize": 6136, "storeFileIndexSize": 1048, "staticIndexSize": 190, "staticBloomSize": 4, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 1974, "localBytesRead": 1974, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803968, "blockCacheCount": 0, "blockCacheDataBlockCount": 0, "blockCacheSize": 601920, "blockCacheCountHitPercent": 0.0, "blockCacheExpressHitPercent": 0.0, "l1CacheSize": 601920, "l1CacheFreeSize": 819803968, "l1CacheCount": 0, "l1CacheEvictionCount": 0, "l1CacheHitCount": 0, "l1CacheMissCount": 0, "l1CacheHitRatio": 0.0, "l1CacheMissRatio": 0.0, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.2, "writeRequestRatePerSecond": 1.4, "ByteBuffAllocatorHeapAllocationBytes": 4608, "ByteBuffAllocatorPoolAllocationBytes": 66560, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 8, "totalRowActionRequestCount": 8, "readRequestCount": 1, "filteredReadRequestCount": 0, "writeRequestCount": 7, "rpcGetRequestCount": 1, "rpcFullScanRequestCount": 0, "rpcScanRequestCount": 0, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 7, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 0, "blockCacheHitCountPrimary": 0, "blockCacheHitCachingCount": 0, "blockCacheMissCount": 0, "blockCacheMissCountPrimary": 0, "blockCacheMissCachingCount": 0, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 0, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 0, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 14, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 1370, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 0, "ScanTime_min": 0, "ScanTime_max": 0, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 0, "ScanTime_75th_percentile": 0, "ScanTime_90th_percentile": 0, "ScanTime_95th_percentile": 0, "ScanTime_98th_percentile": 0, "ScanTime_99th_percentile": 0, "ScanTime_99.9th_percentile": 0, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 7, "Put_min": 1, "Put_max": 4, "Put_mean": 1, "Put_25th_percentile": 1, "Put_median": 2, "Put_75th_percentile": 2, "Put_90th_percentile": 4, "Put_95th_percentile": 4, "Put_98th_percentile": 4, "Put_99th_percentile": 4, "Put_99.9th_percentile": 4, "Put_TimeRangeCount_0-1": 3, "Put_TimeRangeCount_1-3": 3, "Put_TimeRangeCount_3-10": 1, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 2, "FlushMemstoreSize_min": 78, "FlushMemstoreSize_max": 1292, "FlushMemstoreSize_mean": 685, "FlushMemstoreSize_25th_percentile": 78, "FlushMemstoreSize_median": 79, "FlushMemstoreSize_75th_percentile": 774, "FlushMemstoreSize_90th_percentile": 1084, "FlushMemstoreSize_95th_percentile": 1188, "FlushMemstoreSize_98th_percentile": 1250, "FlushMemstoreSize_99th_percentile": 1271, "FlushMemstoreSize_99.9th_percentile": 1289, "FlushMemstoreSize_SizeRangeCount_10-100": 1, "FlushMemstoreSize_SizeRangeCount_100-1000": 1, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 1, "Get_min": 6, "Get_max": 6, "Get_mean": 6, "Get_25th_percentile": 6, "Get_median": 6, "Get_75th_percentile": 6, "Get_90th_percentile": 6, "Get_95th_percentile": 6, "Get_98th_percentile": 6, "Get_99th_percentile": 6, "Get_99.9th_percentile": 6, "Get_TimeRangeCount_3-10": 1, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 15699, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 2, "FlushTime_min": 45, "FlushTime_max": 144, "FlushTime_mean": 94, "FlushTime_25th_percentile": 45, "FlushTime_median": 46, "FlushTime_75th_percentile": 144, "FlushTime_90th_percentile": 144, "FlushTime_95th_percentile": 144, "FlushTime_98th_percentile": 144, "FlushTime_99th_percentile": 144, "FlushTime_99.9th_percentile": 144, "FlushTime_TimeRangeCount_30-100": 1, "FlushTime_TimeRangeCount_100-300": 1, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 0, "ScanSize_min": 0, "ScanSize_max": 0, "ScanSize_mean": 0, "ScanSize_25th_percentile": 0, "ScanSize_median": 0, "ScanSize_75th_percentile": 0, "ScanSize_90th_percentile": 0, "ScanSize_95th_percentile": 0, "ScanSize_98th_percentile": 0, "ScanSize_99th_percentile": 0, "ScanSize_99.9th_percentile": 0, "slowGetCount": 0, "flushedMemstoreBytes": 1370, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 2, "FlushOutputSize_min": 4926, "FlushOutputSize_max": 10773, "FlushOutputSize_mean": 7849, "FlushOutputSize_25th_percentile": 6387, "FlushOutputSize_median": 7849, "FlushOutputSize_75th_percentile": 9311, "FlushOutputSize_90th_percentile": 10188, "FlushOutputSize_95th_percentile": 10480, "FlushOutputSize_98th_percentile": 10656, "FlushOutputSize_99th_percentile": 10714, "FlushOutputSize_99.9th_percentile": 10767, "FlushOutputSize_SizeRangeCount_100-1000": 2, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 14, "ServerReadQueryPerSecond_mean_rate": 1.1973298067803817, "ServerReadQueryPerSecond_1min_rate": 2.408106595110376, "ServerReadQueryPerSecond_5min_rate": 2.560331489171882, "ServerReadQueryPerSecond_15min_rate": 2.5867036352117525, "ServerWriteQueryPerSecond_count": 14, "ServerWriteQueryPerSecond_mean_rate": 1.1973298071899812, "ServerWriteQueryPerSecond_1min_rate": 1.4, "ServerWriteQueryPerSecond_5min_rate": 1.4, "ServerWriteQueryPerSecond_15min_rate": 1.4 } ] } 2023-07-12 00:31:55,696 WARN [RS-EventLoopGroup-5-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase20.apache.org/148.251.75.209:40891 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:40891 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:31:55,696 DEBUG [RS-EventLoopGroup-5-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase20.apache.org/148.251.75.209:40891 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:40891 2023-07-12 00:31:55,697 WARN [RS-EventLoopGroup-5-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase20.apache.org/148.251.75.209:40891 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:40891 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:31:55,697 WARN [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2576): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.net.ConnectException: Call to address=jenkins-hbase20.apache.org/148.251.75.209:40891 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:40891 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:334) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:581) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:16481) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2573) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:196) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$200(MiniHBaseCluster.java:123) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:189) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:186) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2597) at org.apache.hadoop.hbase.MiniHBaseCluster.killAll(MiniHBaseCluster.java:913) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:108) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) Caused by: java.net.ConnectException: Call to address=jenkins-hbase20.apache.org/148.251.75.209:40891 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:40891 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:186) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:385) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:99) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:398) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:368) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1428) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:396) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:913) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:195) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$300(NettyRpcConnection.java:76) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:296) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:287) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:674) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:693) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:40891 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:31:55,698 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:31:55,698 DEBUG [RS:1;jenkins-hbase20:42559] regionserver.HRegionServer(2738): No master found; retry 2023-07-12 00:31:55,698 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,35377,1689121901202 2023-07-12 00:31:55,700 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:31:55,698 DEBUG [RS-EventLoopGroup-5-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase20.apache.org/148.251.75.209:40891 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:40891 2023-07-12 00:31:55,698 DEBUG [M:1;jenkins-hbase20:40891] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,40891,1689121902393 already deleted, retry=false 2023-07-12 00:31:55,700 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,42559,1689121902528' ***** 2023-07-12 00:31:55,700 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2311): STOPPED: killAll 2023-07-12 00:31:55,700 INFO [Listener at localhost.localdomain/34481] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-12 00:31:55,701 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d0e3117 to 127.0.0.1:58965 2023-07-12 00:31:55,701 DEBUG [Listener at localhost.localdomain/34481] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:31:55,702 DEBUG [Listener at localhost.localdomain/34481] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-12 00:31:55,736 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 00:31:55,808 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRegionServerSessionExpired_1689121907370' 2023-07-12 00:31:55,900 INFO [RS:1;jenkins-hbase20:42559] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 00:31:55,900 INFO [RS:1;jenkins-hbase20:42559] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-12 00:31:55,900 INFO [RS:1;jenkins-hbase20:42559] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-12 00:31:55,900 INFO [RS:1;jenkins-hbase20:42559] regionserver.HRegionServer(3305): Received CLOSE for cc777a4cbddcbb1865eb503ec47f25b1 2023-07-12 00:31:55,901 INFO [RS:1;jenkins-hbase20:42559] regionserver.HRegionServer(3305): Received CLOSE for 150e8dd2bd05831664bbcb60e0b3f011 2023-07-12 00:31:55,901 INFO [RS:1;jenkins-hbase20:42559] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,42559,1689121902528 2023-07-12 00:31:55,902 DEBUG [RS:1;jenkins-hbase20:42559] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f14fd09 to 127.0.0.1:58965 2023-07-12 00:31:55,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing cc777a4cbddcbb1865eb503ec47f25b1, disabling compactions & flushes 2023-07-12 00:31:55,903 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region testRegionServerSessionExpired_1689121907370,,1689121907394.cc777a4cbddcbb1865eb503ec47f25b1. 2023-07-12 00:31:55,902 DEBUG [RS:1;jenkins-hbase20:42559] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:31:55,903 INFO [RS:1;jenkins-hbase20:42559] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 00:31:55,903 INFO [RS:1;jenkins-hbase20:42559] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 00:31:55,903 INFO [RS:1;jenkins-hbase20:42559] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 00:31:55,903 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRegionServerSessionExpired_1689121907370,,1689121907394.cc777a4cbddcbb1865eb503ec47f25b1. 2023-07-12 00:31:55,903 INFO [RS:1;jenkins-hbase20:42559] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-12 00:31:55,903 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on testRegionServerSessionExpired_1689121907370,,1689121907394.cc777a4cbddcbb1865eb503ec47f25b1. after waiting 0 ms 2023-07-12 00:31:55,904 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region testRegionServerSessionExpired_1689121907370,,1689121907394.cc777a4cbddcbb1865eb503ec47f25b1. 2023-07-12 00:31:55,904 INFO [RS:1;jenkins-hbase20:42559] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-12 00:31:55,904 DEBUG [RS:1;jenkins-hbase20:42559] regionserver.HRegionServer(1478): Online Regions={cc777a4cbddcbb1865eb503ec47f25b1=testRegionServerSessionExpired_1689121907370,,1689121907394.cc777a4cbddcbb1865eb503ec47f25b1., 1588230740=hbase:meta,,1.1588230740, 150e8dd2bd05831664bbcb60e0b3f011=hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011.} 2023-07-12 00:31:55,905 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 00:31:55,905 DEBUG [RS:1;jenkins-hbase20:42559] regionserver.HRegionServer(1504): Waiting on 150e8dd2bd05831664bbcb60e0b3f011, 1588230740, cc777a4cbddcbb1865eb503ec47f25b1 2023-07-12 00:31:55,906 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 00:31:55,906 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1825): Memstore data size is 41 in region testRegionServerSessionExpired_1689121907370,,1689121907394.cc777a4cbddcbb1865eb503ec47f25b1. 2023-07-12 00:31:55,906 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 00:31:55,907 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 00:31:55,907 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 00:31:55,908 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed testRegionServerSessionExpired_1689121907370,,1689121907394.cc777a4cbddcbb1865eb503ec47f25b1. 2023-07-12 00:31:55,908 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for cc777a4cbddcbb1865eb503ec47f25b1: 2023-07-12 00:31:55,908 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed testRegionServerSessionExpired_1689121907370,,1689121907394.cc777a4cbddcbb1865eb503ec47f25b1. 2023-07-12 00:31:55,910 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 150e8dd2bd05831664bbcb60e0b3f011, disabling compactions & flushes 2023-07-12 00:31:55,910 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011. 2023-07-12 00:31:55,910 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011. 2023-07-12 00:31:55,910 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011. after waiting 0 ms 2023-07-12 00:31:55,910 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011. 2023-07-12 00:31:55,913 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011. 2023-07-12 00:31:55,913 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 150e8dd2bd05831664bbcb60e0b3f011: 2023-07-12 00:31:55,914 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1825): Memstore data size is 2711 in region hbase:meta,,1.1588230740 2023-07-12 00:31:55,914 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689121904400.150e8dd2bd05831664bbcb60e0b3f011. 2023-07-12 00:31:55,914 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 00:31:55,914 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 00:31:55,914 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 00:31:55,914 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-12 00:31:55,915 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-12 00:31:55,916 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-12 00:31:55,935 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 00:31:55,974 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:31:55,974 INFO [M:1;jenkins-hbase20:40891] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,40891,1689121902393; zookeeper connection closed. 2023-07-12 00:31:55,974 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:40891-0x1007e9741810001, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:31:56,075 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:31:56,075 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:35377-0x1007e9741810000, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:31:56,075 INFO [M:0;jenkins-hbase20:35377] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,35377,1689121901202; zookeeper connection closed. 2023-07-12 00:31:56,106 INFO [RS:1;jenkins-hbase20:42559] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,42559,1689121902528; all regions closed. 2023-07-12 00:31:56,122 DEBUG [RS:1;jenkins-hbase20:42559] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:31:56,122 INFO [RS:1;jenkins-hbase20:42559] regionserver.LeaseManager(133): Closed leases 2023-07-12 00:31:56,123 INFO [RS:1;jenkins-hbase20:42559] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-12 00:31:56,123 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 00:31:56,123 INFO [RS:1;jenkins-hbase20:42559] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:42559 2023-07-12 00:31:56,126 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:42559-0x1007e9741810003, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,42559,1689121902528 2023-07-12 00:31:56,126 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:42559-0x1007e9741810003, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:31:56,275 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:42559-0x1007e9741810003, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:31:56,276 INFO [RS:1;jenkins-hbase20:42559] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,42559,1689121902528; zookeeper connection closed. 2023-07-12 00:31:56,276 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:42559-0x1007e9741810003, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:31:56,278 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@18052ecf] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@18052ecf 2023-07-12 00:31:56,278 INFO [Listener at localhost.localdomain/34481] util.JVMClusterUtil(335): Shutdown of 2 master(s) and 2 regionserver(s) complete 2023-07-12 00:31:56,279 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:31:56,285 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=testing utility connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:31:56,289 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility0x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:31:56,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): testing utility-0x1007e9741810009 connected 2023-07-12 00:31:56,290 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2023-07-12 00:31:56,291 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs 2023-07-12 00:31:56,292 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/splitWAL 2023-07-12 00:31:56,292 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters 2023-07-12 00:31:56,293 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc 2023-07-12 00:31:56,294 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master-maintenance 2023-07-12 00:31:56,294 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot 2023-07-12 00:31:56,295 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/switch 2023-07-12 00:31:56,296 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-12 00:31:56,297 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/draining 2023-07-12 00:31:56,298 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-07-12 00:31:56,299 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/hbaseid 2023-07-12 00:31:56,300 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/table 2023-07-12 00:31:56,303 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/abort 2023-07-12 00:31:56,303 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired 2023-07-12 00:31:56,304 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached 2023-07-12 00:31:56,305 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/abort 2023-07-12 00:31:56,305 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired 2023-07-12 00:31:56,305 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached 2023-07-12 00:31:56,307 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/default 2023-07-12 00:31:56,307 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/hbase 2023-07-12 00:31:56,308 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/table/hbase:meta 2023-07-12 00:31:56,309 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/table/hbase:namespace 2023-07-12 00:31:56,309 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/table/testRegionServerSessionExpired_1689121907370 2023-07-12 00:31:56,321 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table/testRegionServerSessionExpired_1689121907370 2023-07-12 00:31:56,322 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/table 2023-07-12 00:31:56,323 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table/hbase:namespace 2023-07-12 00:31:56,323 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table/hbase:meta 2023-07-12 00:31:56,323 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 00:31:56,323 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 00:31:56,323 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 00:31:56,323 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached 2023-07-12 00:31:56,323 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot 2023-07-12 00:31:56,323 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 00:31:56,323 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 00:31:56,323 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached 2023-07-12 00:31:56,323 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc 2023-07-12 00:31:56,323 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 00:31:56,323 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 00:31:56,323 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table 2023-07-12 00:31:56,323 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:56,324 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/hbaseid 2023-07-12 00:31:56,324 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace 2023-07-12 00:31:56,324 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/draining 2023-07-12 00:31:56,324 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 00:31:56,324 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/switch 2023-07-12 00:31:56,324 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot 2023-07-12 00:31:56,324 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master-maintenance 2023-07-12 00:31:56,324 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc 2023-07-12 00:31:56,324 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:31:56,324 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL 2023-07-12 00:31:56,324 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs 2023-07-12 00:31:56,324 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 00:31:56,324 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase 2023-07-12 00:31:56,340 INFO [Listener at localhost.localdomain/34481] hbase.ResourceChecker(175): after: TestZooKeeper#testRegionServerSessionExpired Thread=263 (was 208) Potentially hanging thread: RpcClient-timer-pool-0 java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:600) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:496) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase20:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: Abort regionserver monitor java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58965@0x6f14fd09-SendThread(127.0.0.1:58965) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:228) org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1338) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1261) Potentially hanging thread: LeaseRenewer:jenkins.hfs.0@localhost.localdomain:44295 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: NIOWorkerThread-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: org.apache.hadoop.hdfs.PeerCache@15a502c7 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:253) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: NIOWorkerThread-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: master/jenkins-hbase20:0:becomeActiveMaster-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins@localhost.localdomain:44295 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: NIOWorkerThread-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Monitor thread for TaskMonitor java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.monitoring.TaskMonitor$MonitorRunnable.run(TaskMonitor.java:327) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: NIOWorkerThread-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: master/jenkins-hbase20:0:becomeActiveMaster-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase20:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: IPC Client (1879248547) connection to localhost.localdomain/127.0.0.1:44295 from jenkins.hfs.1 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: NIOWorkerThread-11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: NIOWorkerThread-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: NIOWorkerThread-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.1@localhost.localdomain:44295 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: SnapshotHandlerChoreCleaner sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: NIOWorkerThread-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Abort regionserver monitor java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/cluster_2bc97589-c177-c63b-7f92-2672baa4b2fe/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: NIOWorkerThread-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HBase-Metrics2-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: zk-event-processor-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1879248547) connection to /127.0.0.1:34417 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: ReplicationExecutor-0 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:704) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: NIOWorkerThread-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58965@0x6f14fd09-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: NIOWorkerThread-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58965@0x6f14fd09 java.lang.Object.wait(Native Method) java.lang.Thread.join(Thread.java:1257) java.lang.Thread.join(Thread.java:1331) org.apache.zookeeper.ClientCnxn.disconnect(ClientCnxn.java:1455) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1485) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$51/84184242.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: NIOWorkerThread-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost.localdomain/34481-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: region-location-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/cluster_2bc97589-c177-c63b-7f92-2672baa4b2fe/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: NIOWorkerThread-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-1 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: NIOWorkerThread-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: NIOWorkerThread-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost.localdomain/34481-SendThread(127.0.0.1:58965) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: Idle-Rpc-Conn-Sweeper-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=610 (was 471) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=182 (was 163) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 173), AvailableMemoryMB=4657 (was 5182) 2023-07-12 00:31:56,351 INFO [Listener at localhost.localdomain/34481] hbase.ResourceChecker(147): before: TestZooKeeper#testMasterSessionExpired Thread=263, OpenFileDescriptor=610, MaxFileDescriptor=60000, SystemLoadAverage=182, ProcessCount=172, AvailableMemoryMB=4655 2023-07-12 00:31:56,767 INFO [Listener at localhost.localdomain/34481] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983 with version=8 2023-07-12 00:31:56,767 INFO [Listener at localhost.localdomain/34481] hbase.HBaseTestingUtility(1408): The hbase.fs.tmp.dir is set to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/hbase-staging 2023-07-12 00:31:56,772 INFO [Listener at localhost.localdomain/34481] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-07-12 00:31:56,772 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:31:56,773 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 00:31:56,773 INFO [Listener at localhost.localdomain/34481] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 00:31:56,773 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:31:56,774 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 00:31:56,774 INFO [Listener at localhost.localdomain/34481] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 00:31:56,776 INFO [Listener at localhost.localdomain/34481] ipc.NettyRpcServer(120): Bind to /148.251.75.209:33137 2023-07-12 00:31:56,776 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:31:56,778 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:31:56,779 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=master:33137 connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:31:56,781 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:331370x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:31:56,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:33137-0x1007e974181000a connected 2023-07-12 00:31:56,791 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:31:56,791 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 00:31:56,792 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 00:31:56,794 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33137 2023-07-12 00:31:56,794 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33137 2023-07-12 00:31:56,798 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33137 2023-07-12 00:31:56,802 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33137 2023-07-12 00:31:56,803 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33137 2023-07-12 00:31:56,803 INFO [Listener at localhost.localdomain/34481] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983, hbase.cluster.distributed=false 2023-07-12 00:31:56,805 INFO [Listener at localhost.localdomain/34481] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-07-12 00:31:56,806 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:31:56,806 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 00:31:56,806 INFO [Listener at localhost.localdomain/34481] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 00:31:56,806 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:31:56,806 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 00:31:56,806 INFO [Listener at localhost.localdomain/34481] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 00:31:56,807 INFO [Listener at localhost.localdomain/34481] ipc.NettyRpcServer(120): Bind to /148.251.75.209:38283 2023-07-12 00:31:56,808 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:31:56,809 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:31:56,809 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=master:38283 connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:31:56,814 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:382830x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:31:56,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:38283-0x1007e974181000b connected 2023-07-12 00:31:56,818 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:31:56,818 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 00:31:56,818 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 00:31:56,819 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38283 2023-07-12 00:31:56,819 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38283 2023-07-12 00:31:56,819 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38283 2023-07-12 00:31:56,819 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38283 2023-07-12 00:31:56,819 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38283 2023-07-12 00:31:56,820 INFO [Listener at localhost.localdomain/34481] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983, hbase.cluster.distributed=false 2023-07-12 00:31:56,836 INFO [Listener at localhost.localdomain/34481] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-07-12 00:31:56,836 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:31:56,836 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 00:31:56,836 INFO [Listener at localhost.localdomain/34481] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 00:31:56,836 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:31:56,836 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 00:31:56,836 INFO [Listener at localhost.localdomain/34481] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 00:31:56,838 INFO [Listener at localhost.localdomain/34481] ipc.NettyRpcServer(120): Bind to /148.251.75.209:36343 2023-07-12 00:31:56,838 INFO [Listener at localhost.localdomain/34481] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 00:31:56,839 DEBUG [Listener at localhost.localdomain/34481] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 00:31:56,839 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:31:56,840 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:31:56,841 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36343 connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:31:56,844 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:363430x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:31:56,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:36343-0x1007e974181000c connected 2023-07-12 00:31:56,845 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:36343-0x1007e974181000c, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:31:56,846 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:36343-0x1007e974181000c, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 00:31:56,846 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:36343-0x1007e974181000c, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 00:31:56,847 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36343 2023-07-12 00:31:56,847 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36343 2023-07-12 00:31:56,847 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36343 2023-07-12 00:31:56,847 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36343 2023-07-12 00:31:56,848 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36343 2023-07-12 00:31:56,859 INFO [Listener at localhost.localdomain/34481] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-07-12 00:31:56,859 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:31:56,859 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 00:31:56,859 INFO [Listener at localhost.localdomain/34481] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 00:31:56,859 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:31:56,859 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 00:31:56,859 INFO [Listener at localhost.localdomain/34481] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 00:31:56,861 INFO [Listener at localhost.localdomain/34481] ipc.NettyRpcServer(120): Bind to /148.251.75.209:36703 2023-07-12 00:31:56,861 INFO [Listener at localhost.localdomain/34481] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 00:31:56,862 DEBUG [Listener at localhost.localdomain/34481] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 00:31:56,863 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:31:56,864 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:31:56,864 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36703 connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:31:56,867 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:367030x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:31:56,868 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:367030x0, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:31:56,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:36703-0x1007e974181000d connected 2023-07-12 00:31:56,869 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:36703-0x1007e974181000d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 00:31:56,870 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:36703-0x1007e974181000d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 00:31:56,870 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36703 2023-07-12 00:31:56,871 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36703 2023-07-12 00:31:56,871 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36703 2023-07-12 00:31:56,872 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36703 2023-07-12 00:31:56,872 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36703 2023-07-12 00:31:56,874 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,38283,1689121916805 2023-07-12 00:31:56,874 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,33137,1689121916771 2023-07-12 00:31:56,881 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:31:56,881 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:31:56,881 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,38283,1689121916805 2023-07-12 00:31:56,881 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,33137,1689121916771 2023-07-12 00:31:56,882 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:36343-0x1007e974181000c, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:31:56,882 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:36703-0x1007e974181000d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:31:56,882 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:31:56,884 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:56,884 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:31:56,884 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:56,884 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:31:56,885 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:31:56,886 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,38283,1689121916805 from backup master directory 2023-07-12 00:31:56,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:31:56,886 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(278): Another master is the active master, jenkins-hbase20.apache.org,38283,1689121916805; waiting to become the next active master 2023-07-12 00:31:56,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:31:56,887 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:31:56,887 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 00:31:56,887 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,38283,1689121916805 2023-07-12 00:31:56,887 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,38283,1689121916805 2023-07-12 00:31:56,887 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:31:56,906 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/hbase.id with ID: 074eae06-36d2-46d0-af82-f632d9e8eed4 2023-07-12 00:31:56,927 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:31:56,930 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:56,930 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:56,943 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x7515b01d to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:31:56,947 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4ecd9a23, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:31:56,947 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 00:31:56,948 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 00:31:56,948 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:31:56,950 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store-tmp 2023-07-12 00:31:56,965 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:31:56,965 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 00:31:56,965 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:31:56,965 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:31:56,965 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 00:31:56,965 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:31:56,965 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:31:56,965 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 00:31:56,967 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,38283,1689121916805 2023-07-12 00:31:56,970 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C38283%2C1689121916805, suffix=, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,38283,1689121916805, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/oldWALs, maxLogs=10 2023-07-12 00:31:56,985 DEBUG [RS-EventLoopGroup-10-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:31:56,986 DEBUG [RS-EventLoopGroup-10-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:31:56,993 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,38283,1689121916805/jenkins-hbase20.apache.org%2C38283%2C1689121916805.1689121916971 2023-07-12 00:31:56,994 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK], DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK]] 2023-07-12 00:31:56,994 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 00:31:56,994 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:31:56,994 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:31:56,994 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:31:56,997 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:31:56,999 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 00:31:56,999 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 00:31:57,000 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:31:57,001 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:31:57,002 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:31:57,005 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:31:57,007 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:31:57,008 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74388800, jitterRate=0.10847949981689453}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:31:57,008 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 00:31:57,008 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 00:31:57,010 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-12 00:31:57,010 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-12 00:31:57,010 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 00:31:57,010 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 00:31:57,011 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-07-12 00:31:57,011 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 00:31:57,011 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-12 00:31:57,012 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-12 00:31:57,023 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 00:31:57,024 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 00:31:57,024 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 00:31:57,024 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 00:31:57,026 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:57,026 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:57,026 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 00:31:57,027 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 00:31:57,027 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 00:31:57,028 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:36703-0x1007e974181000d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 00:31:57,028 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 00:31:57,028 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 00:31:57,028 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:36343-0x1007e974181000c, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 00:31:57,028 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:57,028 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:57,029 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,38283,1689121916805, sessionid=0x1007e974181000b, setting cluster-up flag (Was=false) 2023-07-12 00:31:57,032 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:57,032 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:57,032 INFO [M:0;jenkins-hbase20:33137] regionserver.HRegionServer(951): ClusterId : 074eae06-36d2-46d0-af82-f632d9e8eed4 2023-07-12 00:31:57,036 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 00:31:57,037 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,38283,1689121916805 2023-07-12 00:31:57,039 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:57,039 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:57,042 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 00:31:57,043 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,38283,1689121916805 2023-07-12 00:31:57,044 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.hbase-snapshot/.tmp 2023-07-12 00:31:57,047 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-12 00:31:57,047 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:31:57,047 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:31:57,047 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:31:57,048 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:31:57,048 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-07-12 00:31:57,048 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:57,048 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-12 00:31:57,048 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:57,049 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689121947049 2023-07-12 00:31:57,049 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 00:31:57,050 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 00:31:57,050 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 00:31:57,050 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 00:31:57,050 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 00:31:57,050 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 00:31:57,051 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:57,052 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 00:31:57,052 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 00:31:57,052 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 00:31:57,052 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 00:31:57,052 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-12 00:31:57,053 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 00:31:57,053 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 00:31:57,053 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689121917053,5,FailOnTimeoutGroup] 2023-07-12 00:31:57,053 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689121917053,5,FailOnTimeoutGroup] 2023-07-12 00:31:57,053 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:57,054 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 00:31:57,054 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:57,054 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:57,055 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 00:31:57,074 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 00:31:57,075 INFO [RS:0;jenkins-hbase20:36343] regionserver.HRegionServer(951): ClusterId : 074eae06-36d2-46d0-af82-f632d9e8eed4 2023-07-12 00:31:57,076 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 00:31:57,076 DEBUG [RS:0;jenkins-hbase20:36343] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 00:31:57,075 INFO [RS:1;jenkins-hbase20:36703] regionserver.HRegionServer(951): ClusterId : 074eae06-36d2-46d0-af82-f632d9e8eed4 2023-07-12 00:31:57,077 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983 2023-07-12 00:31:57,078 DEBUG [RS:1;jenkins-hbase20:36703] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 00:31:57,080 DEBUG [RS:0;jenkins-hbase20:36343] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 00:31:57,080 DEBUG [RS:0;jenkins-hbase20:36343] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 00:31:57,081 DEBUG [RS:1;jenkins-hbase20:36703] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 00:31:57,081 DEBUG [RS:1;jenkins-hbase20:36703] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 00:31:57,084 DEBUG [RS:0;jenkins-hbase20:36343] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 00:31:57,085 DEBUG [RS:1;jenkins-hbase20:36703] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 00:31:57,087 DEBUG [RS:0;jenkins-hbase20:36343] zookeeper.ReadOnlyZKClient(139): Connect 0x2afa529e to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:31:57,087 DEBUG [RS:1;jenkins-hbase20:36703] zookeeper.ReadOnlyZKClient(139): Connect 0x787c4f9b to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:31:57,099 DEBUG [RS:0;jenkins-hbase20:36343] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@243db9b1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:31:57,099 DEBUG [RS:1;jenkins-hbase20:36703] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7c43d44c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:31:57,100 DEBUG [RS:0;jenkins-hbase20:36343] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@200239ee, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-12 00:31:57,100 DEBUG [RS:1;jenkins-hbase20:36703] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@394145a0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-12 00:31:57,101 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:31:57,103 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 00:31:57,105 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/info 2023-07-12 00:31:57,106 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 00:31:57,106 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:31:57,107 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 00:31:57,109 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/rep_barrier 2023-07-12 00:31:57,109 DEBUG [RS:0;jenkins-hbase20:36343] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:36343 2023-07-12 00:31:57,110 INFO [RS:0;jenkins-hbase20:36343] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 00:31:57,110 INFO [RS:0;jenkins-hbase20:36343] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 00:31:57,110 DEBUG [RS:0;jenkins-hbase20:36343] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 00:31:57,110 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 00:31:57,110 DEBUG [RS:1;jenkins-hbase20:36703] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase20:36703 2023-07-12 00:31:57,111 INFO [RS:1;jenkins-hbase20:36703] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 00:31:57,111 INFO [RS:1;jenkins-hbase20:36703] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 00:31:57,111 DEBUG [RS:1;jenkins-hbase20:36703] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 00:31:57,111 INFO [RS:0;jenkins-hbase20:36343] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase20.apache.org,38283,1689121916805 with isa=jenkins-hbase20.apache.org/148.251.75.209:36343, startcode=1689121916835 2023-07-12 00:31:57,111 DEBUG [RS:0;jenkins-hbase20:36343] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 00:31:57,112 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:31:57,112 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 00:31:57,113 INFO [RS:1;jenkins-hbase20:36703] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase20.apache.org,38283,1689121916805 with isa=jenkins-hbase20.apache.org/148.251.75.209:36703, startcode=1689121916858 2023-07-12 00:31:57,114 DEBUG [RS:1;jenkins-hbase20:36703] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 00:31:57,115 INFO [RS-EventLoopGroup-8-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:33921, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 00:31:57,116 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/table 2023-07-12 00:31:57,117 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38283] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,36343,1689121916835 2023-07-12 00:31:57,117 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 00:31:57,118 INFO [RS-EventLoopGroup-8-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:53851, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 00:31:57,118 DEBUG [RS:0;jenkins-hbase20:36343] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983 2023-07-12 00:31:57,119 DEBUG [RS:0;jenkins-hbase20:36343] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:44295 2023-07-12 00:31:57,119 DEBUG [RS:0;jenkins-hbase20:36343] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 00:31:57,119 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:31:57,119 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38283] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,36703,1689121916858 2023-07-12 00:31:57,121 DEBUG [RS:1;jenkins-hbase20:36703] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983 2023-07-12 00:31:57,121 DEBUG [RS:1;jenkins-hbase20:36703] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:44295 2023-07-12 00:31:57,121 DEBUG [RS:1;jenkins-hbase20:36703] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 00:31:57,121 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740 2023-07-12 00:31:57,122 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740 2023-07-12 00:31:57,122 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:31:57,122 DEBUG [RS:0;jenkins-hbase20:36343] zookeeper.ZKUtil(162): regionserver:36343-0x1007e974181000c, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,36343,1689121916835 2023-07-12 00:31:57,123 WARN [RS:0;jenkins-hbase20:36343] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 00:31:57,123 INFO [RS:0;jenkins-hbase20:36343] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:31:57,123 DEBUG [RS:0;jenkins-hbase20:36343] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,36343,1689121916835 2023-07-12 00:31:57,124 DEBUG [RS:1;jenkins-hbase20:36703] zookeeper.ZKUtil(162): regionserver:36703-0x1007e974181000d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,36703,1689121916858 2023-07-12 00:31:57,124 WARN [RS:1;jenkins-hbase20:36703] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 00:31:57,124 INFO [RS:1;jenkins-hbase20:36703] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:31:57,124 DEBUG [RS:1;jenkins-hbase20:36703] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,36703,1689121916858 2023-07-12 00:31:57,124 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,36343,1689121916835] 2023-07-12 00:31:57,124 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,36703,1689121916858] 2023-07-12 00:31:57,130 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 00:31:57,131 DEBUG [RS:0;jenkins-hbase20:36343] zookeeper.ZKUtil(162): regionserver:36343-0x1007e974181000c, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,36343,1689121916835 2023-07-12 00:31:57,131 DEBUG [RS:1;jenkins-hbase20:36703] zookeeper.ZKUtil(162): regionserver:36703-0x1007e974181000d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,36343,1689121916835 2023-07-12 00:31:57,131 DEBUG [RS:0;jenkins-hbase20:36343] zookeeper.ZKUtil(162): regionserver:36343-0x1007e974181000c, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,36703,1689121916858 2023-07-12 00:31:57,131 DEBUG [RS:1;jenkins-hbase20:36703] zookeeper.ZKUtil(162): regionserver:36703-0x1007e974181000d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,36703,1689121916858 2023-07-12 00:31:57,132 DEBUG [RS:0;jenkins-hbase20:36343] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 00:31:57,132 DEBUG [RS:1;jenkins-hbase20:36703] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 00:31:57,132 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 00:31:57,132 INFO [RS:0;jenkins-hbase20:36343] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 00:31:57,132 INFO [RS:1;jenkins-hbase20:36703] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 00:31:57,134 INFO [RS:0;jenkins-hbase20:36343] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 00:31:57,134 INFO [RS:0;jenkins-hbase20:36343] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 00:31:57,134 INFO [RS:0;jenkins-hbase20:36343] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:57,135 INFO [RS:0;jenkins-hbase20:36343] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 00:31:57,136 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:31:57,136 INFO [RS:1;jenkins-hbase20:36703] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 00:31:57,136 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69045845, jitterRate=0.028863266110420227}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 00:31:57,136 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 00:31:57,136 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 00:31:57,137 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 00:31:57,137 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 00:31:57,137 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 00:31:57,137 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 00:31:57,137 INFO [RS:1;jenkins-hbase20:36703] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 00:31:57,137 INFO [RS:1;jenkins-hbase20:36703] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:57,142 INFO [RS:0;jenkins-hbase20:36343] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:57,142 INFO [RS:1;jenkins-hbase20:36703] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 00:31:57,143 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 00:31:57,143 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 00:31:57,143 DEBUG [RS:0;jenkins-hbase20:36343] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:57,143 DEBUG [RS:0;jenkins-hbase20:36343] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:57,143 DEBUG [RS:0;jenkins-hbase20:36343] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:57,143 DEBUG [RS:0;jenkins-hbase20:36343] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:57,143 DEBUG [RS:0;jenkins-hbase20:36343] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:57,144 DEBUG [RS:0;jenkins-hbase20:36343] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-12 00:31:57,144 DEBUG [RS:0;jenkins-hbase20:36343] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:57,144 DEBUG [RS:0;jenkins-hbase20:36343] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:57,144 DEBUG [RS:0;jenkins-hbase20:36343] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:57,144 DEBUG [RS:0;jenkins-hbase20:36343] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:57,144 INFO [RS:1;jenkins-hbase20:36703] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:57,145 INFO [RS:0;jenkins-hbase20:36343] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:57,146 DEBUG [RS:1;jenkins-hbase20:36703] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:57,146 INFO [RS:0;jenkins-hbase20:36343] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:57,146 DEBUG [RS:1;jenkins-hbase20:36703] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:57,146 INFO [RS:0;jenkins-hbase20:36343] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:57,146 DEBUG [RS:1;jenkins-hbase20:36703] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:57,146 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 00:31:57,146 DEBUG [RS:1;jenkins-hbase20:36703] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:57,146 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-12 00:31:57,146 DEBUG [RS:1;jenkins-hbase20:36703] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:57,146 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-12 00:31:57,146 DEBUG [RS:1;jenkins-hbase20:36703] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-12 00:31:57,146 DEBUG [RS:1;jenkins-hbase20:36703] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:57,146 DEBUG [RS:1;jenkins-hbase20:36703] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:57,146 DEBUG [RS:1;jenkins-hbase20:36703] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:57,147 DEBUG [RS:1;jenkins-hbase20:36703] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:31:57,150 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-12 00:31:57,154 INFO [RS:1;jenkins-hbase20:36703] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:57,155 INFO [RS:1;jenkins-hbase20:36703] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:57,155 INFO [RS:1;jenkins-hbase20:36703] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:57,155 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-12 00:31:57,164 INFO [RS:1;jenkins-hbase20:36703] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 00:31:57,164 INFO [RS:1;jenkins-hbase20:36703] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,36703,1689121916858-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:57,165 INFO [RS:0;jenkins-hbase20:36343] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 00:31:57,166 INFO [RS:0;jenkins-hbase20:36343] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,36343,1689121916835-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:57,174 INFO [RS:1;jenkins-hbase20:36703] regionserver.Replication(203): jenkins-hbase20.apache.org,36703,1689121916858 started 2023-07-12 00:31:57,174 INFO [RS:1;jenkins-hbase20:36703] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,36703,1689121916858, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:36703, sessionid=0x1007e974181000d 2023-07-12 00:31:57,174 DEBUG [RS:1;jenkins-hbase20:36703] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 00:31:57,174 DEBUG [RS:1;jenkins-hbase20:36703] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,36703,1689121916858 2023-07-12 00:31:57,174 DEBUG [RS:1;jenkins-hbase20:36703] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,36703,1689121916858' 2023-07-12 00:31:57,174 DEBUG [RS:1;jenkins-hbase20:36703] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 00:31:57,175 DEBUG [RS:1;jenkins-hbase20:36703] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 00:31:57,175 DEBUG [RS:1;jenkins-hbase20:36703] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 00:31:57,175 DEBUG [RS:1;jenkins-hbase20:36703] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 00:31:57,175 DEBUG [RS:1;jenkins-hbase20:36703] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,36703,1689121916858 2023-07-12 00:31:57,175 DEBUG [RS:1;jenkins-hbase20:36703] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,36703,1689121916858' 2023-07-12 00:31:57,175 DEBUG [RS:1;jenkins-hbase20:36703] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 00:31:57,176 DEBUG [RS:1;jenkins-hbase20:36703] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 00:31:57,176 DEBUG [RS:1;jenkins-hbase20:36703] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 00:31:57,176 INFO [RS:1;jenkins-hbase20:36703] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 00:31:57,176 INFO [RS:1;jenkins-hbase20:36703] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 00:31:57,182 INFO [RS:0;jenkins-hbase20:36343] regionserver.Replication(203): jenkins-hbase20.apache.org,36343,1689121916835 started 2023-07-12 00:31:57,182 INFO [RS:0;jenkins-hbase20:36343] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,36343,1689121916835, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:36343, sessionid=0x1007e974181000c 2023-07-12 00:31:57,183 DEBUG [RS:0;jenkins-hbase20:36343] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 00:31:57,183 DEBUG [RS:0;jenkins-hbase20:36343] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,36343,1689121916835 2023-07-12 00:31:57,183 DEBUG [RS:0;jenkins-hbase20:36343] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,36343,1689121916835' 2023-07-12 00:31:57,183 DEBUG [RS:0;jenkins-hbase20:36343] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 00:31:57,183 DEBUG [RS:0;jenkins-hbase20:36343] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 00:31:57,184 DEBUG [RS:0;jenkins-hbase20:36343] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 00:31:57,184 DEBUG [RS:0;jenkins-hbase20:36343] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 00:31:57,184 DEBUG [RS:0;jenkins-hbase20:36343] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,36343,1689121916835 2023-07-12 00:31:57,184 DEBUG [RS:0;jenkins-hbase20:36343] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,36343,1689121916835' 2023-07-12 00:31:57,184 DEBUG [RS:0;jenkins-hbase20:36343] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 00:31:57,184 DEBUG [RS:0;jenkins-hbase20:36343] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 00:31:57,185 DEBUG [RS:0;jenkins-hbase20:36343] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 00:31:57,185 INFO [RS:0;jenkins-hbase20:36343] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 00:31:57,185 INFO [RS:0;jenkins-hbase20:36343] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 00:31:57,279 INFO [RS:1;jenkins-hbase20:36703] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C36703%2C1689121916858, suffix=, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,36703,1689121916858, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/oldWALs, maxLogs=32 2023-07-12 00:31:57,288 INFO [RS:0;jenkins-hbase20:36343] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C36343%2C1689121916835, suffix=, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,36343,1689121916835, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/oldWALs, maxLogs=32 2023-07-12 00:31:57,300 DEBUG [RS-EventLoopGroup-10-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:31:57,300 DEBUG [RS-EventLoopGroup-10-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:31:57,303 INFO [RS:1;jenkins-hbase20:36703] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,36703,1689121916858/jenkins-hbase20.apache.org%2C36703%2C1689121916858.1689121917281 2023-07-12 00:31:57,303 DEBUG [RS:1;jenkins-hbase20:36703] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK], DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK]] 2023-07-12 00:31:57,306 DEBUG [jenkins-hbase20:38283] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-12 00:31:57,309 DEBUG [jenkins-hbase20:38283] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-07-12 00:31:57,312 DEBUG [RS-EventLoopGroup-10-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:31:57,312 DEBUG [RS-EventLoopGroup-10-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:31:57,313 DEBUG [jenkins-hbase20:38283] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 00:31:57,313 DEBUG [jenkins-hbase20:38283] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 00:31:57,313 DEBUG [jenkins-hbase20:38283] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 00:31:57,314 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,36703,1689121916858, state=OPENING 2023-07-12 00:31:57,315 INFO [RS:0;jenkins-hbase20:36343] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,36343,1689121916835/jenkins-hbase20.apache.org%2C36343%2C1689121916835.1689121917289 2023-07-12 00:31:57,315 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-12 00:31:57,315 DEBUG [RS:0;jenkins-hbase20:36343] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK], DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK]] 2023-07-12 00:31:57,316 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:57,316 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:57,316 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 00:31:57,316 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,36703,1689121916858}] 2023-07-12 00:31:57,317 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 00:31:57,472 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,36703,1689121916858 2023-07-12 00:31:57,472 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 00:31:57,481 INFO [RS-EventLoopGroup-10-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50310, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 00:31:57,490 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-12 00:31:57,490 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:31:57,493 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C36703%2C1689121916858.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,36703,1689121916858, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/oldWALs, maxLogs=32 2023-07-12 00:31:57,507 DEBUG [RS-EventLoopGroup-10-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:31:57,508 DEBUG [RS-EventLoopGroup-10-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:31:57,510 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,36703,1689121916858/jenkins-hbase20.apache.org%2C36703%2C1689121916858.meta.1689121917494.meta 2023-07-12 00:31:57,511 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK], DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK]] 2023-07-12 00:31:57,511 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-12 00:31:57,511 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-12 00:31:57,511 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-12 00:31:57,511 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-12 00:31:57,511 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-12 00:31:57,511 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:31:57,511 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-12 00:31:57,512 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-12 00:31:57,513 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 00:31:57,514 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/info 2023-07-12 00:31:57,514 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/info 2023-07-12 00:31:57,515 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 00:31:57,516 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:31:57,516 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 00:31:57,517 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/rep_barrier 2023-07-12 00:31:57,517 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/rep_barrier 2023-07-12 00:31:57,517 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 00:31:57,518 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:31:57,518 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 00:31:57,519 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/table 2023-07-12 00:31:57,519 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/table 2023-07-12 00:31:57,520 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 00:31:57,521 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:31:57,522 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740 2023-07-12 00:31:57,525 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740 2023-07-12 00:31:57,528 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 00:31:57,531 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 00:31:57,532 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59141707, jitterRate=-0.11871989071369171}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 00:31:57,532 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 00:31:57,533 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689121917472 2023-07-12 00:31:57,538 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-12 00:31:57,539 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-12 00:31:57,539 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,36703,1689121916858, state=OPEN 2023-07-12 00:31:57,541 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 00:31:57,541 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 00:31:57,541 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 00:31:57,541 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 00:31:57,544 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-12 00:31:57,544 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,36703,1689121916858 in 225 msec 2023-07-12 00:31:57,546 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-12 00:31:57,546 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 398 msec 2023-07-12 00:31:57,548 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 501 msec 2023-07-12 00:31:57,548 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689121917548, completionTime=-1 2023-07-12 00:31:57,548 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-12 00:31:57,548 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 00:31:57,551 DEBUG [hconnection-0x715c4fe8-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 00:31:57,553 INFO [RS-EventLoopGroup-10-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50324, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 00:31:57,554 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-12 00:31:57,554 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689121977554 2023-07-12 00:31:57,554 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689122037554 2023-07-12 00:31:57,554 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 6 msec 2023-07-12 00:31:57,562 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,38283,1689121916805-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:57,562 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,38283,1689121916805-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:57,562 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,38283,1689121916805-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:57,562 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:38283, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:57,562 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 00:31:57,562 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-12 00:31:57,562 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 00:31:57,563 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-12 00:31:57,564 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 00:31:57,566 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 00:31:57,567 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 00:31:57,570 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/hbase/namespace/a1d134596bd623a40abe345dfeab7ab9 2023-07-12 00:31:57,570 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/hbase/namespace/a1d134596bd623a40abe345dfeab7ab9 empty. 2023-07-12 00:31:57,571 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/hbase/namespace/a1d134596bd623a40abe345dfeab7ab9 2023-07-12 00:31:57,571 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-12 00:31:57,588 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-12 00:31:57,590 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => a1d134596bd623a40abe345dfeab7ab9, NAME => 'hbase:namespace,,1689121917562.a1d134596bd623a40abe345dfeab7ab9.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp 2023-07-12 00:31:57,604 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689121917562.a1d134596bd623a40abe345dfeab7ab9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:31:57,605 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing a1d134596bd623a40abe345dfeab7ab9, disabling compactions & flushes 2023-07-12 00:31:57,605 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689121917562.a1d134596bd623a40abe345dfeab7ab9. 2023-07-12 00:31:57,605 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689121917562.a1d134596bd623a40abe345dfeab7ab9. 2023-07-12 00:31:57,605 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689121917562.a1d134596bd623a40abe345dfeab7ab9. after waiting 0 ms 2023-07-12 00:31:57,605 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689121917562.a1d134596bd623a40abe345dfeab7ab9. 2023-07-12 00:31:57,605 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689121917562.a1d134596bd623a40abe345dfeab7ab9. 2023-07-12 00:31:57,605 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for a1d134596bd623a40abe345dfeab7ab9: 2023-07-12 00:31:57,608 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 00:31:57,610 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689121917562.a1d134596bd623a40abe345dfeab7ab9.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689121917609"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689121917609"}]},"ts":"1689121917609"} 2023-07-12 00:31:57,613 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 00:31:57,614 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 00:31:57,614 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689121917614"}]},"ts":"1689121917614"} 2023-07-12 00:31:57,616 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-12 00:31:57,618 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-07-12 00:31:57,619 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 00:31:57,619 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 00:31:57,619 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 00:31:57,619 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=a1d134596bd623a40abe345dfeab7ab9, ASSIGN}] 2023-07-12 00:31:57,622 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=a1d134596bd623a40abe345dfeab7ab9, ASSIGN 2023-07-12 00:31:57,623 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=a1d134596bd623a40abe345dfeab7ab9, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36343,1689121916835; forceNewPlan=false, retain=false 2023-07-12 00:31:57,773 INFO [jenkins-hbase20:38283] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-12 00:31:57,776 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=a1d134596bd623a40abe345dfeab7ab9, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36343,1689121916835 2023-07-12 00:31:57,777 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689121917562.a1d134596bd623a40abe345dfeab7ab9.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689121917776"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121917776"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121917776"}]},"ts":"1689121917776"} 2023-07-12 00:31:57,781 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure a1d134596bd623a40abe345dfeab7ab9, server=jenkins-hbase20.apache.org,36343,1689121916835}] 2023-07-12 00:31:57,938 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,36343,1689121916835 2023-07-12 00:31:57,938 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 00:31:57,943 INFO [RS-EventLoopGroup-9-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47368, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 00:31:57,951 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689121917562.a1d134596bd623a40abe345dfeab7ab9. 2023-07-12 00:31:57,951 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a1d134596bd623a40abe345dfeab7ab9, NAME => 'hbase:namespace,,1689121917562.a1d134596bd623a40abe345dfeab7ab9.', STARTKEY => '', ENDKEY => ''} 2023-07-12 00:31:57,952 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace a1d134596bd623a40abe345dfeab7ab9 2023-07-12 00:31:57,953 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689121917562.a1d134596bd623a40abe345dfeab7ab9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:31:57,953 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for a1d134596bd623a40abe345dfeab7ab9 2023-07-12 00:31:57,953 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for a1d134596bd623a40abe345dfeab7ab9 2023-07-12 00:31:57,955 INFO [StoreOpener-a1d134596bd623a40abe345dfeab7ab9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region a1d134596bd623a40abe345dfeab7ab9 2023-07-12 00:31:57,957 DEBUG [StoreOpener-a1d134596bd623a40abe345dfeab7ab9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/a1d134596bd623a40abe345dfeab7ab9/info 2023-07-12 00:31:57,957 DEBUG [StoreOpener-a1d134596bd623a40abe345dfeab7ab9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/a1d134596bd623a40abe345dfeab7ab9/info 2023-07-12 00:31:57,957 INFO [StoreOpener-a1d134596bd623a40abe345dfeab7ab9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a1d134596bd623a40abe345dfeab7ab9 columnFamilyName info 2023-07-12 00:31:57,958 INFO [StoreOpener-a1d134596bd623a40abe345dfeab7ab9-1] regionserver.HStore(310): Store=a1d134596bd623a40abe345dfeab7ab9/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:31:57,959 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/a1d134596bd623a40abe345dfeab7ab9 2023-07-12 00:31:57,960 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/a1d134596bd623a40abe345dfeab7ab9 2023-07-12 00:31:57,964 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for a1d134596bd623a40abe345dfeab7ab9 2023-07-12 00:31:57,969 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/a1d134596bd623a40abe345dfeab7ab9/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:31:57,970 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened a1d134596bd623a40abe345dfeab7ab9; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68408451, jitterRate=0.019365355372428894}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:31:57,970 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for a1d134596bd623a40abe345dfeab7ab9: 2023-07-12 00:31:57,971 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689121917562.a1d134596bd623a40abe345dfeab7ab9., pid=6, masterSystemTime=1689121917937 2023-07-12 00:31:57,977 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689121917562.a1d134596bd623a40abe345dfeab7ab9. 2023-07-12 00:31:57,978 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689121917562.a1d134596bd623a40abe345dfeab7ab9. 2023-07-12 00:31:57,979 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=a1d134596bd623a40abe345dfeab7ab9, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36343,1689121916835 2023-07-12 00:31:57,979 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689121917562.a1d134596bd623a40abe345dfeab7ab9.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689121917978"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121917978"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121917978"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121917978"}]},"ts":"1689121917978"} 2023-07-12 00:31:57,986 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-12 00:31:57,986 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure a1d134596bd623a40abe345dfeab7ab9, server=jenkins-hbase20.apache.org,36343,1689121916835 in 202 msec 2023-07-12 00:31:57,989 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-12 00:31:57,990 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=a1d134596bd623a40abe345dfeab7ab9, ASSIGN in 367 msec 2023-07-12 00:31:57,991 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 00:31:57,991 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689121917991"}]},"ts":"1689121917991"} 2023-07-12 00:31:57,993 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-12 00:31:57,995 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 00:31:57,997 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 433 msec 2023-07-12 00:31:58,065 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-12 00:31:58,067 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:58,067 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-12 00:31:58,067 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:58,074 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 00:31:58,076 INFO [RS-EventLoopGroup-9-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47374, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 00:31:58,079 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-12 00:31:58,091 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 00:31:58,096 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 16 msec 2023-07-12 00:31:58,102 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-12 00:31:58,115 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 00:31:58,120 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 16 msec 2023-07-12 00:31:58,131 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 00:31:58,132 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 00:31:58,133 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 1.245sec 2023-07-12 00:31:58,133 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 00:31:58,133 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 00:31:58,133 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 00:31:58,133 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,38283,1689121916805-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 00:31:58,133 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,38283,1689121916805-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 00:31:58,136 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 00:31:58,177 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ReadOnlyZKClient(139): Connect 0x50cb0c5c to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:31:58,188 DEBUG [Listener at localhost.localdomain/34481] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a5b58bd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:31:58,191 DEBUG [hconnection-0x700a735-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 00:31:58,195 INFO [RS-EventLoopGroup-10-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50326, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 00:31:58,197 INFO [Listener at localhost.localdomain/34481] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,38283,1689121916805 2023-07-12 00:31:58,197 INFO [Listener at localhost.localdomain/34481] hbase.TestZooKeeper(128): Starting testMasterSessionExpired 2023-07-12 00:31:58,202 INFO [Listener at localhost.localdomain/34481-EventThread] hbase.HBaseTestingUtility$2(2877): Monitor ZKW received event=WatchedEvent state:SyncConnected type:None path:null 2023-07-12 00:31:58,204 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:31:58,204 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:36343-0x1007e974181000c, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:31:58,204 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:58,204 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:36703-0x1007e974181000d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:31:58,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:31:58,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36343-0x1007e974181000c, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:31:58,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36703-0x1007e974181000d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:31:58,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:31:58,205 DEBUG [zk-event-processor-pool-0] master.ActiveMasterManager(186): No master available. Notifying waiting threads 2023-07-12 00:31:58,206 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:36703-0x1007e974181000d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:31:58,206 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:31:58,206 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:36343-0x1007e974181000c, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:31:58,206 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:31:58,206 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:31:58,207 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,33137,1689121916771 from backup master directory 2023-07-12 00:31:58,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:31:58,208 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,33137,1689121916771 2023-07-12 00:31:58,208 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:31:58,208 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 00:31:58,208 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,33137,1689121916771 2023-07-12 00:31:58,227 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:31:58,257 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x21880936 to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:31:58,261 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1cca9422, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:31:58,262 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 00:31:58,263 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 00:31:58,263 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:31:58,273 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,38283,1689121916805 to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,38283,1689121916805-dead as it is dead 2023-07-12 00:31:58,273 INFO [master/jenkins-hbase20:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,38283,1689121916805-dead/jenkins-hbase20.apache.org%2C38283%2C1689121916805.1689121916971 2023-07-12 00:31:58,274 WARN [IPC Server handler 4 on default port 44295] namenode.FSNamesystem(3291): DIR* NameSystem.internalReleaseLease: File /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,38283,1689121916805-dead/jenkins-hbase20.apache.org%2C38283%2C1689121916805.1689121916971 has not been closed. Lease recovery is in progress. RecoveryId = 1035 for block blk_1073741849_1026 2023-07-12 00:31:58,275 INFO [master/jenkins-hbase20:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Failed to recover lease, attempt=0 on file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,38283,1689121916805-dead/jenkins-hbase20.apache.org%2C38283%2C1689121916805.1689121916971 after 1ms 2023-07-12 00:31:58,301 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2023-07-12 00:31:58,301 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2023-07-12 00:31:58,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(630): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2023-07-12 00:31:58,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(630): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2023-07-12 00:31:58,304 INFO [Listener at localhost.localdomain/34481] hbase.HBaseTestingUtility(2892): ZK Closed Session 0x1007e974181000b 2023-07-12 00:31:59,359 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Expired, path=null 2023-07-12 00:31:59,359 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Expired, path=null 2023-07-12 00:31:59,359 ERROR [zk-event-processor-pool-0] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 00:31:59,360 ERROR [zk-event-processor-pool-0] helpers.MarkerIgnoringBase(159): ***** ABORTING master jenkins-hbase20.apache.org,38283,1689121916805: master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase master:38283-0x1007e974181000b received expired from ZooKeeper, aborting ***** org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.zookeeper.ZKWatcher.connectionEvent(ZKWatcher.java:642) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.processEvent(ZKWatcher.java:555) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.lambda$process$0(ZKWatcher.java:603) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:31:59,361 INFO [zk-event-processor-pool-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,38283,1689121916805' ***** 2023-07-12 00:31:59,361 INFO [zk-event-processor-pool-0] regionserver.HRegionServer(2311): STOPPED: Stopped by zk-event-processor-pool-0 2023-07-12 00:31:59,361 WARN [zk-event-processor-pool-0] zookeeper.ZKUtil(402): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Unable to get data of znode /hbase/master org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:134) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:2131) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:330) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:394) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:145) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:324) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2824) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2816) at org.apache.hadoop.hbase.master.HMaster.abort(HMaster.java:2727) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.connectionEvent(ZKWatcher.java:642) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.processEvent(ZKWatcher.java:555) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.lambda$process$0(ZKWatcher.java:603) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:31:59,361 DEBUG [M:1;jenkins-hbase20:38283] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@634615b6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-12 00:31:59,361 ERROR [zk-event-processor-pool-0] zookeeper.ZKWatcher(701): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:134) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:2131) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:330) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:394) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:145) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:324) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2824) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2816) at org.apache.hadoop.hbase.master.HMaster.abort(HMaster.java:2727) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.connectionEvent(ZKWatcher.java:642) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.processEvent(ZKWatcher.java:555) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.lambda$process$0(ZKWatcher.java:603) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:31:59,361 INFO [M:1;jenkins-hbase20:38283] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,38283,1689121916805 2023-07-12 00:31:59,362 DEBUG [zk-event-processor-pool-0] master.ActiveMasterManager(335): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Failed delete of our master address node; KeeperErrorCode = Session expired for /hbase/master 2023-07-12 00:31:59,362 DEBUG [zk-event-processor-pool-0] master.HMaster(2711): Abort called but aborted=true, stopped=true 2023-07-12 00:31:59,362 DEBUG [M:1;jenkins-hbase20:38283] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7515b01d to 127.0.0.1:58965 2023-07-12 00:31:59,362 DEBUG [M:1;jenkins-hbase20:38283] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:31:59,362 INFO [M:1;jenkins-hbase20:38283] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,38283,1689121916805; all regions closed. 2023-07-12 00:31:59,362 DEBUG [M:1;jenkins-hbase20:38283] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:31:59,362 DEBUG [M:1;jenkins-hbase20:38283] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 00:31:59,363 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 00:31:59,364 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689121917053] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689121917053,5,FailOnTimeoutGroup] 2023-07-12 00:31:59,363 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689121917053] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689121917053,5,FailOnTimeoutGroup] 2023-07-12 00:31:59,363 DEBUG [M:1;jenkins-hbase20:38283] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 00:31:59,365 INFO [M:1;jenkins-hbase20:38283] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 00:31:59,365 INFO [M:1;jenkins-hbase20:38283] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 00:31:59,366 INFO [M:1;jenkins-hbase20:38283] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-07-12 00:31:59,366 DEBUG [M:1;jenkins-hbase20:38283] master.HMaster(1512): Stopping service threads 2023-07-12 00:31:59,366 INFO [M:1;jenkins-hbase20:38283] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 00:31:59,366 INFO [M:1;jenkins-hbase20:38283] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 00:31:59,368 ERROR [M:1;jenkins-hbase20:38283] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-3,5,PEWorkerGroup] 2023-07-12 00:31:59,368 INFO [M:1;jenkins-hbase20:38283] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=true 2023-07-12 00:31:59,368 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 00:31:59,565 WARN [M:1;jenkins-hbase20:38283] zookeeper.ZKUtil(402): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Unable to get data of znode /hbase/master org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:134) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:2131) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:330) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:394) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:145) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:324) at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1527) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1205) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:541) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:31:59,566 ERROR [M:1;jenkins-hbase20:38283] zookeeper.ZKWatcher(701): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:134) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:2131) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:330) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:394) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:145) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:324) at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1527) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1205) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:541) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:31:59,566 DEBUG [M:1;jenkins-hbase20:38283] master.ActiveMasterManager(335): master:38283-0x1007e974181000b, quorum=127.0.0.1:58965, baseZNode=/hbase Failed delete of our master address node; KeeperErrorCode = Session expired for /hbase/master 2023-07-12 00:31:59,566 INFO [M:1;jenkins-hbase20:38283] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 00:31:59,567 INFO [M:1;jenkins-hbase20:38283] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=true 2023-07-12 00:31:59,571 WARN [IPC Server handler 1 on default port 44295] ipc.Server(2892): IPC Server handler 1 on default port 44295, call Call#894 Retry#0 org.apache.hadoop.hdfs.protocol.ClientProtocol.complete from 127.0.0.1:42538 java.lang.AssertionError: Negative replicas! at org.apache.hadoop.hdfs.server.blockmanagement.UnderReplicatedBlocks.getPriority(UnderReplicatedBlocks.java:154) at org.apache.hadoop.hdfs.server.blockmanagement.UnderReplicatedBlocks.update(UnderReplicatedBlocks.java:323) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.updateNeededReplications(BlockManager.java:3774) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.commitOrCompleteLastBlock(BlockManager.java:870) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.commitOrCompleteLastBlock(FSNamesystem.java:3325) at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.completeFileInternal(FSDirWriteFileOp.java:649) at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.completeFile(FSDirWriteFileOp.java:601) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2741) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:935) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:562) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:507) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1034) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:994) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:922) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2833) 2023-07-12 00:31:59,573 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(649): complete file /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,38283,1689121916805/jenkins-hbase20.apache.org%2C38283%2C1689121916805.1689121916971 failed, retry = 0 org.apache.hadoop.ipc.RemoteException(java.lang.AssertionError): Negative replicas! at org.apache.hadoop.hdfs.server.blockmanagement.UnderReplicatedBlocks.getPriority(UnderReplicatedBlocks.java:154) at org.apache.hadoop.hdfs.server.blockmanagement.UnderReplicatedBlocks.update(UnderReplicatedBlocks.java:323) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.updateNeededReplications(BlockManager.java:3774) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.commitOrCompleteLastBlock(BlockManager.java:870) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.commitOrCompleteLastBlock(FSNamesystem.java:3325) at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.completeFileInternal(FSDirWriteFileOp.java:649) at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.completeFile(FSDirWriteFileOp.java:601) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2741) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:935) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:562) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:507) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1034) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:994) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:922) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2833) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1540) at org.apache.hadoop.ipc.Client.call(Client.java:1486) at org.apache.hadoop.ipc.Client.call(Client.java:1385) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy31.complete(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:488) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy34.complete(Unknown Source) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy35.complete(Unknown Source) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy35.complete(Unknown Source) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy35.complete(Unknown Source) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy35.complete(Unknown Source) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy35.complete(Unknown Source) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy35.complete(Unknown Source) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy35.complete(Unknown Source) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy35.complete(Unknown Source) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy35.complete(Unknown Source) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy35.complete(Unknown Source) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy35.complete(Unknown Source) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy35.complete(Unknown Source) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy35.complete(Unknown Source) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.completeFile(FanOutOneBlockAsyncDFSOutputHelper.java:637) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:586) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:169) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:746) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:31:59,676 DEBUG [M:1;jenkins-hbase20:38283] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 00:31:59,677 INFO [M:1;jenkins-hbase20:38283] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:31:59,677 DEBUG [M:1;jenkins-hbase20:38283] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:31:59,677 DEBUG [M:1;jenkins-hbase20:38283] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 00:31:59,677 DEBUG [M:1;jenkins-hbase20:38283] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:31:59,677 ERROR [M:1;jenkins-hbase20:38283] regionserver.HRegion(1825): Memstore data size is 24669 in region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:31:59,678 INFO [M:1;jenkins-hbase20:38283] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:31:59,678 DEBUG [M:1;jenkins-hbase20:38283] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 00:31:59,678 INFO [M:1;jenkins-hbase20:38283] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 00:31:59,678 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 00:31:59,679 INFO [M:1;jenkins-hbase20:38283] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:38283 2023-07-12 00:31:59,684 WARN [M:1;jenkins-hbase20:38283] regionserver.HRegionServer(1217): Failed deleting my ephemeral node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/jenkins-hbase20.apache.org,38283,1689121916805 at org.apache.zookeeper.KeeperException.create(KeeperException.java:134) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:1793) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:162) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:902) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:894) at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1694) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1213) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:541) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:31:59,686 INFO [M:1;jenkins-hbase20:38283] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,38283,1689121916805; zookeeper connection closed. 2023-07-12 00:31:59,765 WARN [RS-EventLoopGroup-10-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase20.apache.org/148.251.75.209:38283 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:38283 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:31:59,766 DEBUG [RS-EventLoopGroup-10-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase20.apache.org/148.251.75.209:38283 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:38283 2023-07-12 00:31:59,768 WARN [RS-EventLoopGroup-10-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase20.apache.org/148.251.75.209:38283 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:38283 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:31:59,769 DEBUG [RS-EventLoopGroup-10-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase20.apache.org/148.251.75.209:38283 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:38283 2023-07-12 00:31:59,867 DEBUG [RS:1;jenkins-hbase20:36703] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 00:31:59,869 INFO [RS-EventLoopGroup-7-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:45333, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 00:31:59,870 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:31:59,870 DEBUG [RS:0;jenkins-hbase20:36343] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 00:31:59,872 INFO [RS-EventLoopGroup-7-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:56869, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 00:31:59,872 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:31:59,972 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:31:59,974 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:00,075 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:00,076 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:00,179 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:00,180 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:00,284 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:00,284 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:00,388 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:00,388 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:00,491 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:00,491 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:00,594 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:00,594 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:00,698 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:00,699 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:00,804 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:00,804 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:00,906 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:00,908 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:00,970 INFO [Listener at localhost.localdomain/34481-EventThread] hbase.HBaseTestingUtility$2(2877): Monitor ZKW received event=WatchedEvent state:Closed type:None path:null 2023-07-12 00:32:00,971 INFO [Listener at localhost.localdomain/34481] hbase.TestZooKeeper(154): Creating table testMasterSessionExpired_1689121920970 2023-07-12 00:32:00,974 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-12 00:32:00,976 INFO [RS-EventLoopGroup-7-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40212, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-12 00:32:00,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:1166) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:01,074 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:01,074 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:01,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:1166) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:01,177 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:01,177 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:01,198 WARN [BP-1731987413-148.251.75.209-1689121899843 heartbeating to localhost.localdomain/127.0.0.1:44295] datanode.BPServiceActor(725): RemoteException in offerService org.apache.hadoop.ipc.RemoteException(java.io.IOException): Recovery block blk_1073741849_1026where it is not under construction. at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.getBlockRecoveryCommand(DatanodeManager.java:1479) at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.handleHeartbeat(DatanodeManager.java:1587) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:3733) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.sendHeartbeat(NameNodeRpcServer.java:1480) at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.sendHeartbeat(DatanodeProtocolServerSideTranslatorPB.java:119) at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:30585) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:507) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1034) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:994) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:922) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2833) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1540) at org.apache.hadoop.ipc.Client.call(Client.java:1486) at org.apache.hadoop.ipc.Client.call(Client.java:1385) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy30.sendHeartbeat(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:166) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:516) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:646) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:32:01,280 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:01,280 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:01,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:1166) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:01,383 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:01,384 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:01,413 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 00:32:01,487 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:01,488 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:01,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:1166) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:01,591 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:01,591 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:01,698 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:01,698 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:01,801 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:01,802 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:01,904 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:01,904 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:02,007 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:02,007 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:02,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:1166) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:02,111 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:02,112 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:02,216 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:02,217 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:02,276 INFO [master/jenkins-hbase20:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=1 on file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,38283,1689121916805-dead/jenkins-hbase20.apache.org%2C38283%2C1689121916805.1689121916971 after 4002ms 2023-07-12 00:32:02,277 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,38283,1689121916805-dead/jenkins-hbase20.apache.org%2C38283%2C1689121916805.1689121916971 to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase20.apache.org%2C38283%2C1689121916805.1689121916971 2023-07-12 00:32:02,277 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,38283,1689121916805-dead 2023-07-12 00:32:02,278 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,33137,1689121916771 2023-07-12 00:32:02,280 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C33137%2C1689121916771, suffix=, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,33137,1689121916771, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/oldWALs, maxLogs=10 2023-07-12 00:32:02,294 DEBUG [RS-EventLoopGroup-10-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:32:02,294 DEBUG [RS-EventLoopGroup-10-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:32:02,296 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,33137,1689121916771/jenkins-hbase20.apache.org%2C33137%2C1689121916771.1689121922281 2023-07-12 00:32:02,298 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK], DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK]] 2023-07-12 00:32:02,298 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 00:32:02,299 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:02,299 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:02,299 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:02,301 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:02,303 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 00:32:02,303 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 00:32:02,304 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:02,305 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5179): Found 1 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-07-12 00:32:02,305 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5276): Replaying edits from hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase20.apache.org%2C38283%2C1689121916805.1689121916971 2023-07-12 00:32:02,315 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5464): Applied 71, skipped 0, firstSequenceIdInLog=3, maxSequenceIdInLog=65, path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase20.apache.org%2C38283%2C1689121916805.1689121916971 2023-07-12 00:32:02,315 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=24.09 KB heapSize=29.59 KB 2023-07-12 00:32:02,321 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:02,321 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33137] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:02,330 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.09 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/b04e9362747349c9886eb55202805261 2023-07-12 00:32:02,338 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/b04e9362747349c9886eb55202805261 as hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/b04e9362747349c9886eb55202805261 2023-07-12 00:32:02,345 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/b04e9362747349c9886eb55202805261, entries=71, sequenceid=65, filesize=28.9 K 2023-07-12 00:32:02,346 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(2948): Finished flush of dataSize ~24.09 KB/24669, heapSize ~29.57 KB/30280, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 31ms, sequenceid=65, compaction requested=false; wal=null 2023-07-12 00:32:02,347 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase20.apache.org%2C38283%2C1689121916805.1689121916971 2023-07-12 00:32:02,351 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:02,355 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/65.seqid, newMaxSeqId=65, maxSeqId=1 2023-07-12 00:32:02,356 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=66; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73003726, jitterRate=0.08784028887748718}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:02,356 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 00:32:02,356 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 00:32:02,358 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-12 00:32:02,358 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-12 00:32:02,358 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 00:32:02,358 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 00:32:02,368 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 00:32:02,369 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 00:32:02,369 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 00:32:02,369 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 00:32:02,369 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 10 msec 2023-07-12 00:32:02,369 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 00:32:02,370 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 00:32:02,370 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase20.apache.org,36703,1689121916858, table=hbase:meta, region=1588230740 2023-07-12 00:32:02,372 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 2 possibly 'live' servers, and 0 'splitting'. 2023-07-12 00:32:02,373 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,36343,1689121916835 2023-07-12 00:32:02,373 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,36703,1689121916858 2023-07-12 00:32:02,386 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 00:32:02,386 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 00:32:02,386 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 00:32:02,387 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 00:32:02,387 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 00:32:02,388 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 00:32:02,388 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 00:32:02,388 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,33137,1689121916771, sessionid=0x1007e974181000a, setting cluster-up flag (Was=true) 2023-07-12 00:32:02,390 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 00:32:02,390 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,33137,1689121916771 2023-07-12 00:32:02,392 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 00:32:02,393 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,33137,1689121916771 2023-07-12 00:32:02,393 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.hbase-snapshot/.tmp 2023-07-12 00:32:02,394 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:32:02,394 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:32:02,394 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:32:02,394 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:32:02,394 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-07-12 00:32:02,395 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:02,395 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-12 00:32:02,395 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:02,397 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689121952397 2023-07-12 00:32:02,397 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 00:32:02,397 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 00:32:02,398 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 00:32:02,398 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 00:32:02,398 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 00:32:02,398 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 00:32:02,398 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:02,399 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 00:32:02,399 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 00:32:02,399 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 00:32:02,399 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 00:32:02,399 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 00:32:02,400 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689121922399,5,FailOnTimeoutGroup] 2023-07-12 00:32:02,402 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689121922400,5,FailOnTimeoutGroup] 2023-07-12 00:32:02,402 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:02,402 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 00:32:02,403 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:02,403 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:02,403 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689121922403, completionTime=-1 2023-07-12 00:32:02,403 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-12 00:32:02,403 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 00:32:02,405 DEBUG [hconnection-0x69464328-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 00:32:02,406 INFO [RS-EventLoopGroup-10-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:45344, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 00:32:02,409 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=a1d134596bd623a40abe345dfeab7ab9, regionState=OPEN, lastHost=jenkins-hbase20.apache.org,36343,1689121916835, regionLocation=jenkins-hbase20.apache.org,36343,1689121916835, openSeqNum=2 2023-07-12 00:32:02,409 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-12 00:32:02,409 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689121982409 2023-07-12 00:32:02,409 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689122042409 2023-07-12 00:32:02,409 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 6 msec 2023-07-12 00:32:02,422 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,33137,1689121916771-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:02,422 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,33137,1689121916771-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:02,422 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,33137,1689121916771-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:02,422 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:33137, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:02,422 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:02,423 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-07-12 00:32:02,428 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 00:32:02,430 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 00:32:02,430 INFO [RS-EventLoopGroup-9-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:33558, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 00:32:02,436 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 00:32:02,442 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 00:32:02,442 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 4.234sec 2023-07-12 00:32:02,442 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 00:32:02,442 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 00:32:02,442 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 00:32:02,442 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,33137,1689121916771-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 00:32:02,443 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,33137,1689121916771-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 00:32:02,445 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 00:32:03,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33137] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'testMasterSessionExpired_1689121920970', {NAME => 'fam', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 00:32:03,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33137] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testMasterSessionExpired_1689121920970 2023-07-12 00:32:03,113 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testMasterSessionExpired_1689121920970 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 00:32:03,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33137] master.MasterRpcServices(700): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "testMasterSessionExpired_1689121920970" procId is: 9 2023-07-12 00:32:03,115 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testMasterSessionExpired_1689121920970 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 00:32:03,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33137] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 00:32:03,118 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testMasterSessionExpired_1689121920970/64f62a3e3514274d6ca189fac5775125 2023-07-12 00:32:03,118 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testMasterSessionExpired_1689121920970/64f62a3e3514274d6ca189fac5775125 empty. 2023-07-12 00:32:03,119 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testMasterSessionExpired_1689121920970/64f62a3e3514274d6ca189fac5775125 2023-07-12 00:32:03,119 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testMasterSessionExpired_1689121920970 regions 2023-07-12 00:32:03,132 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-12 00:32:03,133 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-12 00:32:03,137 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testMasterSessionExpired_1689121920970/.tabledesc/.tableinfo.0000000001 2023-07-12 00:32:03,142 INFO [RegionOpenAndInit-testMasterSessionExpired_1689121920970-pool-0] regionserver.HRegion(7675): creating {ENCODED => 64f62a3e3514274d6ca189fac5775125, NAME => 'testMasterSessionExpired_1689121920970,,1689121923108.64f62a3e3514274d6ca189fac5775125.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testMasterSessionExpired_1689121920970', {NAME => 'fam', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp 2023-07-12 00:32:03,159 DEBUG [RegionOpenAndInit-testMasterSessionExpired_1689121920970-pool-0] regionserver.HRegion(866): Instantiated testMasterSessionExpired_1689121920970,,1689121923108.64f62a3e3514274d6ca189fac5775125.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:03,160 DEBUG [RegionOpenAndInit-testMasterSessionExpired_1689121920970-pool-0] regionserver.HRegion(1604): Closing 64f62a3e3514274d6ca189fac5775125, disabling compactions & flushes 2023-07-12 00:32:03,160 INFO [RegionOpenAndInit-testMasterSessionExpired_1689121920970-pool-0] regionserver.HRegion(1626): Closing region testMasterSessionExpired_1689121920970,,1689121923108.64f62a3e3514274d6ca189fac5775125. 2023-07-12 00:32:03,160 DEBUG [RegionOpenAndInit-testMasterSessionExpired_1689121920970-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testMasterSessionExpired_1689121920970,,1689121923108.64f62a3e3514274d6ca189fac5775125. 2023-07-12 00:32:03,160 DEBUG [RegionOpenAndInit-testMasterSessionExpired_1689121920970-pool-0] regionserver.HRegion(1714): Acquired close lock on testMasterSessionExpired_1689121920970,,1689121923108.64f62a3e3514274d6ca189fac5775125. after waiting 0 ms 2023-07-12 00:32:03,160 DEBUG [RegionOpenAndInit-testMasterSessionExpired_1689121920970-pool-0] regionserver.HRegion(1724): Updates disabled for region testMasterSessionExpired_1689121920970,,1689121923108.64f62a3e3514274d6ca189fac5775125. 2023-07-12 00:32:03,160 INFO [RegionOpenAndInit-testMasterSessionExpired_1689121920970-pool-0] regionserver.HRegion(1838): Closed testMasterSessionExpired_1689121920970,,1689121923108.64f62a3e3514274d6ca189fac5775125. 2023-07-12 00:32:03,160 DEBUG [RegionOpenAndInit-testMasterSessionExpired_1689121920970-pool-0] regionserver.HRegion(1558): Region close journal for 64f62a3e3514274d6ca189fac5775125: 2023-07-12 00:32:03,163 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testMasterSessionExpired_1689121920970 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 00:32:03,164 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testMasterSessionExpired_1689121920970,,1689121923108.64f62a3e3514274d6ca189fac5775125.","families":{"info":[{"qualifier":"regioninfo","vlen":72,"tag":[],"timestamp":"1689121923164"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689121923164"}]},"ts":"1689121923164"} 2023-07-12 00:32:03,166 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 00:32:03,168 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testMasterSessionExpired_1689121920970 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 00:32:03,168 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testMasterSessionExpired_1689121920970","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689121923168"}]},"ts":"1689121923168"} 2023-07-12 00:32:03,170 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testMasterSessionExpired_1689121920970, state=ENABLING in hbase:meta 2023-07-12 00:32:03,172 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-07-12 00:32:03,173 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 00:32:03,173 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 00:32:03,173 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 00:32:03,174 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testMasterSessionExpired_1689121920970, region=64f62a3e3514274d6ca189fac5775125, ASSIGN}] 2023-07-12 00:32:03,176 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testMasterSessionExpired_1689121920970, region=64f62a3e3514274d6ca189fac5775125, ASSIGN 2023-07-12 00:32:03,178 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testMasterSessionExpired_1689121920970, region=64f62a3e3514274d6ca189fac5775125, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36343,1689121916835; forceNewPlan=false, retain=false 2023-07-12 00:32:03,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33137] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 00:32:03,329 INFO [jenkins-hbase20:33137] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-12 00:32:03,330 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=64f62a3e3514274d6ca189fac5775125, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36343,1689121916835 2023-07-12 00:32:03,330 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testMasterSessionExpired_1689121920970,,1689121923108.64f62a3e3514274d6ca189fac5775125.","families":{"info":[{"qualifier":"regioninfo","vlen":72,"tag":[],"timestamp":"1689121923330"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121923330"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121923330"}]},"ts":"1689121923330"} 2023-07-12 00:32:03,332 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 64f62a3e3514274d6ca189fac5775125, server=jenkins-hbase20.apache.org,36343,1689121916835}] 2023-07-12 00:32:03,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33137] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 00:32:03,487 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,36343,1689121916835 2023-07-12 00:32:03,487 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 00:32:03,488 INFO [RS-EventLoopGroup-9-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:33560, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 00:32:03,493 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open testMasterSessionExpired_1689121920970,,1689121923108.64f62a3e3514274d6ca189fac5775125. 2023-07-12 00:32:03,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 64f62a3e3514274d6ca189fac5775125, NAME => 'testMasterSessionExpired_1689121920970,,1689121923108.64f62a3e3514274d6ca189fac5775125.', STARTKEY => '', ENDKEY => ''} 2023-07-12 00:32:03,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testMasterSessionExpired_1689121920970 64f62a3e3514274d6ca189fac5775125 2023-07-12 00:32:03,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated testMasterSessionExpired_1689121920970,,1689121923108.64f62a3e3514274d6ca189fac5775125.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:03,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 64f62a3e3514274d6ca189fac5775125 2023-07-12 00:32:03,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 64f62a3e3514274d6ca189fac5775125 2023-07-12 00:32:03,495 INFO [StoreOpener-64f62a3e3514274d6ca189fac5775125-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family fam of region 64f62a3e3514274d6ca189fac5775125 2023-07-12 00:32:03,497 DEBUG [StoreOpener-64f62a3e3514274d6ca189fac5775125-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testMasterSessionExpired_1689121920970/64f62a3e3514274d6ca189fac5775125/fam 2023-07-12 00:32:03,497 DEBUG [StoreOpener-64f62a3e3514274d6ca189fac5775125-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testMasterSessionExpired_1689121920970/64f62a3e3514274d6ca189fac5775125/fam 2023-07-12 00:32:03,497 INFO [StoreOpener-64f62a3e3514274d6ca189fac5775125-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 64f62a3e3514274d6ca189fac5775125 columnFamilyName fam 2023-07-12 00:32:03,498 INFO [StoreOpener-64f62a3e3514274d6ca189fac5775125-1] regionserver.HStore(310): Store=64f62a3e3514274d6ca189fac5775125/fam, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:03,499 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testMasterSessionExpired_1689121920970/64f62a3e3514274d6ca189fac5775125 2023-07-12 00:32:03,499 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testMasterSessionExpired_1689121920970/64f62a3e3514274d6ca189fac5775125 2023-07-12 00:32:03,502 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 64f62a3e3514274d6ca189fac5775125 2023-07-12 00:32:03,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testMasterSessionExpired_1689121920970/64f62a3e3514274d6ca189fac5775125/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:03,505 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 64f62a3e3514274d6ca189fac5775125; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69815876, jitterRate=0.04033762216567993}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:03,505 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 64f62a3e3514274d6ca189fac5775125: 2023-07-12 00:32:03,506 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testMasterSessionExpired_1689121920970,,1689121923108.64f62a3e3514274d6ca189fac5775125., pid=11, masterSystemTime=1689121923487 2023-07-12 00:32:03,509 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testMasterSessionExpired_1689121920970,,1689121923108.64f62a3e3514274d6ca189fac5775125. 2023-07-12 00:32:03,509 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened testMasterSessionExpired_1689121920970,,1689121923108.64f62a3e3514274d6ca189fac5775125. 2023-07-12 00:32:03,510 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=64f62a3e3514274d6ca189fac5775125, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36343,1689121916835 2023-07-12 00:32:03,510 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testMasterSessionExpired_1689121920970,,1689121923108.64f62a3e3514274d6ca189fac5775125.","families":{"info":[{"qualifier":"regioninfo","vlen":72,"tag":[],"timestamp":"1689121923509"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121923509"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121923509"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121923509"}]},"ts":"1689121923509"} 2023-07-12 00:32:03,515 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-12 00:32:03,515 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 64f62a3e3514274d6ca189fac5775125, server=jenkins-hbase20.apache.org,36343,1689121916835 in 180 msec 2023-07-12 00:32:03,519 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-12 00:32:03,519 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testMasterSessionExpired_1689121920970, region=64f62a3e3514274d6ca189fac5775125, ASSIGN in 342 msec 2023-07-12 00:32:03,520 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testMasterSessionExpired_1689121920970 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 00:32:03,520 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testMasterSessionExpired_1689121920970","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689121923520"}]},"ts":"1689121923520"} 2023-07-12 00:32:03,527 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testMasterSessionExpired_1689121920970, state=ENABLED in hbase:meta 2023-07-12 00:32:03,530 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testMasterSessionExpired_1689121920970 execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 00:32:03,532 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testMasterSessionExpired_1689121920970 in 422 msec 2023-07-12 00:32:03,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33137] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 00:32:03,721 INFO [Listener at localhost.localdomain/34481] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testMasterSessionExpired_1689121920970, procId: 9 completed 2023-07-12 00:32:03,722 INFO [Listener at localhost.localdomain/34481] hbase.TestZooKeeper(165): Putting table testMasterSessionExpired_1689121920970 2023-07-12 00:32:03,726 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 00:32:03,731 INFO [RS-EventLoopGroup-9-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:33566, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 00:32:03,736 DEBUG [Listener at localhost.localdomain/34481] master.HMaster(2711): Abort called but aborted=true, stopped=true 2023-07-12 00:32:03,736 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 00:32:03,736 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): ***** ABORTING master jenkins-hbase20.apache.org,33137,1689121916771: killAll ***** 2023-07-12 00:32:03,736 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,33137,1689121916771' ***** 2023-07-12 00:32:03,737 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2311): STOPPED: Stopped by Listener at localhost.localdomain/34481 2023-07-12 00:32:03,737 DEBUG [M:0;jenkins-hbase20:33137] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1bdd7458, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-12 00:32:03,737 INFO [M:0;jenkins-hbase20:33137] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,33137,1689121916771 2023-07-12 00:32:03,738 DEBUG [M:0;jenkins-hbase20:33137] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21880936 to 127.0.0.1:58965 2023-07-12 00:32:03,738 DEBUG [M:0;jenkins-hbase20:33137] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:03,738 INFO [M:0;jenkins-hbase20:33137] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,33137,1689121916771; all regions closed. 2023-07-12 00:32:03,738 DEBUG [M:0;jenkins-hbase20:33137] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:03,738 DEBUG [M:0;jenkins-hbase20:33137] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 00:32:03,738 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 00:32:03,738 DEBUG [M:0;jenkins-hbase20:33137] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 00:32:03,738 INFO [M:0;jenkins-hbase20:33137] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 00:32:03,738 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689121922400] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689121922400,5,FailOnTimeoutGroup] 2023-07-12 00:32:03,738 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689121922399] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689121922399,5,FailOnTimeoutGroup] 2023-07-12 00:32:03,740 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:36703-0x1007e974181000d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:03,738 INFO [M:0;jenkins-hbase20:33137] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 00:32:03,740 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:36343-0x1007e974181000c, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:03,740 INFO [M:0;jenkins-hbase20:33137] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-07-12 00:32:03,740 DEBUG [M:0;jenkins-hbase20:33137] master.HMaster(1512): Stopping service threads 2023-07-12 00:32:03,740 INFO [M:0;jenkins-hbase20:33137] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 00:32:03,741 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:03,741 INFO [M:0;jenkins-hbase20:33137] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 00:32:03,741 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:03,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36343-0x1007e974181000c, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:03,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36703-0x1007e974181000d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:03,744 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase20.apache.org,36343,1689121916835: killAll ***** 2023-07-12 00:32:03,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:03,745 ERROR [M:0;jenkins-hbase20:33137] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-4,5,PEWorkerGroup] 2023-07-12 00:32:03,744 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 00:32:03,745 INFO [M:0;jenkins-hbase20:33137] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=true 2023-07-12 00:32:03,745 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-12 00:32:03,745 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 00:32:03,745 DEBUG [M:0;jenkins-hbase20:33137] zookeeper.ZKUtil(398): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 00:32:03,745 WARN [M:0;jenkins-hbase20:33137] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 00:32:03,745 INFO [M:0;jenkins-hbase20:33137] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 00:32:03,746 INFO [M:0;jenkins-hbase20:33137] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=true 2023-07-12 00:32:03,746 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-12 00:32:03,752 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-12 00:32:03,776 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-12 00:32:03,779 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1047003136, "init": 524288000, "max": 2051014656, "used": 401307824 }, "NonHeapMemoryUsage": { "committed": 129851392, "init": 2555904, "max": -1, "used": 127545432 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 0, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 13538, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 0, "ProcessCallTime_num_ops": 55, "ProcessCallTime_min": 0, "ProcessCallTime_max": 8, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 2, "ProcessCallTime_75th_percentile": 2, "ProcessCallTime_90th_percentile": 3, "ProcessCallTime_95th_percentile": 7, "ProcessCallTime_98th_percentile": 8, "ProcessCallTime_99th_percentile": 8, "ProcessCallTime_99.9th_percentile": 8, "ProcessCallTime_TimeRangeCount_0-1": 11, "ProcessCallTime_TimeRangeCount_1-3": 9, "ProcessCallTime_TimeRangeCount_3-10": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 55, "QueueCallTime_min": 0, "QueueCallTime_max": 1, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 1, "QueueCallTime_99th_percentile": 1, "QueueCallTime_99.9th_percentile": 1, "QueueCallTime_TimeRangeCount_0-1": 22, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 55, "TotalCallTime_min": 0, "TotalCallTime_max": 8, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 2, "TotalCallTime_75th_percentile": 2, "TotalCallTime_90th_percentile": 3, "TotalCallTime_95th_percentile": 7, "TotalCallTime_98th_percentile": 8, "TotalCallTime_99th_percentile": 8, "TotalCallTime_99.9th_percentile": 8, "TotalCallTime_TimeRangeCount_0-1": 10, "TotalCallTime_TimeRangeCount_1-3": 10, "TotalCallTime_TimeRangeCount_3-10": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 55, "ResponseSize_min": 0, "ResponseSize_max": 983, "ResponseSize_mean": 127, "ResponseSize_25th_percentile": 2, "ResponseSize_median": 4, "ResponseSize_75th_percentile": 22, "ResponseSize_90th_percentile": 449, "ResponseSize_95th_percentile": 716, "ResponseSize_98th_percentile": 876, "ResponseSize_99th_percentile": 929, "ResponseSize_99.9th_percentile": 977, "ResponseSize_SizeRangeCount_0-10": 15, "ResponseSize_SizeRangeCount_10-100": 3, "ResponseSize_SizeRangeCount_100-1000": 4, "exceptions.UnknownScannerException": 0, "exceptions": 0, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 0, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 55, "RequestSize_min": 80, "RequestSize_max": 340, "RequestSize_mean": 143, "RequestSize_25th_percentile": 97, "RequestSize_median": 116, "RequestSize_75th_percentile": 150, "RequestSize_90th_percentile": 216, "RequestSize_95th_percentile": 280, "RequestSize_98th_percentile": 339, "RequestSize_99th_percentile": 339, "RequestSize_99.9th_percentile": 339, "RequestSize_SizeRangeCount_10-100": 8, "RequestSize_SizeRangeCount_100-1000": 14, "sentBytes": 5103 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:58965", "tag.serverName": "jenkins-hbase20.apache.org,42559,1689121902528", "tag.clusterId": "b94c042f-d510-47b7-9d4c-011dde64d627", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "regionCount": 0, "storeCount": 0, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 0, "maxStoreFileCount": 0, "memStoreSize": 0, "storeFileSize": 0, "maxStoreFileAge": 0, "minStoreFileAge": 3067, "avgStoreFileAge": 3537, "numReferenceFiles": 0, "regionServerStartTime": 1689121902528, "averageRegionSize": 6136, "storeFileIndexSize": 0, "staticIndexSize": 0, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 3948, "localBytesRead": 3948, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803968, "blockCacheCount": 0, "blockCacheDataBlockCount": 0, "blockCacheSize": 601920, "blockCacheCountHitPercent": 0.0, "blockCacheExpressHitPercent": 0.0, "l1CacheSize": 601920, "l1CacheFreeSize": 819803968, "l1CacheCount": 0, "l1CacheEvictionCount": 0, "l1CacheHitCount": 0, "l1CacheMissCount": 0, "l1CacheHitRatio": 0.0, "l1CacheMissRatio": 0.0, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 4608, "ByteBuffAllocatorPoolAllocationBytes": 133120, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 10, "totalRowActionRequestCount": 0, "readRequestCount": 0, "filteredReadRequestCount": 0, "writeRequestCount": 0, "rpcGetRequestCount": 1, "rpcFullScanRequestCount": 0, "rpcScanRequestCount": 1, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 8, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 0, "blockCacheHitCountPrimary": 0, "blockCacheHitCachingCount": 0, "blockCacheMissCount": 0, "blockCacheMissCountPrimary": 0, "blockCacheMissCachingCount": 0, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 0, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 0, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 0, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 0, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 1, "ScanTime_min": 1, "ScanTime_max": 1, "ScanTime_mean": 1, "ScanTime_25th_percentile": 1, "ScanTime_median": 1, "ScanTime_75th_percentile": 1, "ScanTime_90th_percentile": 1, "ScanTime_95th_percentile": 1, "ScanTime_98th_percentile": 1, "ScanTime_99th_percentile": 1, "ScanTime_99.9th_percentile": 1, "ScanTime_TimeRangeCount_0-1": 1, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 8, "Put_min": 2, "Put_max": 2, "Put_mean": 2, "Put_25th_percentile": 2, "Put_median": 2, "Put_75th_percentile": 2, "Put_90th_percentile": 2, "Put_95th_percentile": 2, "Put_98th_percentile": 2, "Put_99th_percentile": 2, "Put_99.9th_percentile": 2, "Put_TimeRangeCount_1-3": 1, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 2, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 1, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 15699, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 2, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 1, "ScanSize_min": 956, "ScanSize_max": 956, "ScanSize_mean": 956, "ScanSize_25th_percentile": 956, "ScanSize_median": 956, "ScanSize_75th_percentile": 956, "ScanSize_90th_percentile": 956, "ScanSize_95th_percentile": 956, "ScanSize_98th_percentile": 956, "ScanSize_99th_percentile": 956, "ScanSize_99.9th_percentile": 956, "ScanSize_SizeRangeCount_100-1000": 1, "slowGetCount": 0, "flushedMemstoreBytes": 1370, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 2, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 28, "ServerReadQueryPerSecond_mean_rate": 1.5853034410975209, "ServerReadQueryPerSecond_1min_rate": 2.4394406617012336, "ServerReadQueryPerSecond_5min_rate": 2.5642928612206086, "ServerReadQueryPerSecond_15min_rate": 2.587885329492682, "ServerWriteQueryPerSecond_count": 22, "ServerWriteQueryPerSecond_mean_rate": 1.2455953139609977, "ServerWriteQueryPerSecond_1min_rate": 1.4159911170741353, "ServerWriteQueryPerSecond_5min_rate": 1.4033057092356764, "ServerWriteQueryPerSecond_15min_rate": 1.4011080303990207 } ] } 2023-07-12 00:32:03,780 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33137] master.MasterRpcServices(612): jenkins-hbase20.apache.org,36343,1689121916835 reported a fatal error: ***** ABORTING region server jenkins-hbase20.apache.org,36343,1689121916835: killAll ***** 2023-07-12 00:32:03,783 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,36343,1689121916835' ***** 2023-07-12 00:32:03,783 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2311): STOPPED: killAll 2023-07-12 00:32:03,788 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase20.apache.org,36703,1689121916858: killAll ***** 2023-07-12 00:32:03,788 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 00:32:03,788 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-12 00:32:03,789 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-12 00:32:03,789 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-12 00:32:03,790 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-12 00:32:03,791 INFO [RS:0;jenkins-hbase20:36343] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 00:32:03,791 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 00:32:03,791 INFO [RS:0;jenkins-hbase20:36343] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-12 00:32:03,791 INFO [RS:0;jenkins-hbase20:36343] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-12 00:32:03,791 INFO [RS:0;jenkins-hbase20:36343] regionserver.HRegionServer(3305): Received CLOSE for 64f62a3e3514274d6ca189fac5775125 2023-07-12 00:32:03,792 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1047003136, "init": 524288000, "max": 2051014656, "used": 405955544 }, "NonHeapMemoryUsage": { "committed": 130113536, "init": 2555904, "max": -1, "used": 127751200 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 0, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 13538, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 0, "ProcessCallTime_num_ops": 55, "ProcessCallTime_min": 0, "ProcessCallTime_max": 8, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 2, "ProcessCallTime_75th_percentile": 2, "ProcessCallTime_90th_percentile": 3, "ProcessCallTime_95th_percentile": 7, "ProcessCallTime_98th_percentile": 8, "ProcessCallTime_99th_percentile": 8, "ProcessCallTime_99.9th_percentile": 8, "ProcessCallTime_TimeRangeCount_0-1": 11, "ProcessCallTime_TimeRangeCount_1-3": 9, "ProcessCallTime_TimeRangeCount_3-10": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 55, "QueueCallTime_min": 0, "QueueCallTime_max": 1, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 1, "QueueCallTime_99th_percentile": 1, "QueueCallTime_99.9th_percentile": 1, "QueueCallTime_TimeRangeCount_0-1": 22, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 55, "TotalCallTime_min": 0, "TotalCallTime_max": 8, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 2, "TotalCallTime_75th_percentile": 2, "TotalCallTime_90th_percentile": 3, "TotalCallTime_95th_percentile": 7, "TotalCallTime_98th_percentile": 8, "TotalCallTime_99th_percentile": 8, "TotalCallTime_99.9th_percentile": 8, "TotalCallTime_TimeRangeCount_0-1": 10, "TotalCallTime_TimeRangeCount_1-3": 10, "TotalCallTime_TimeRangeCount_3-10": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 55, "ResponseSize_min": 0, "ResponseSize_max": 983, "ResponseSize_mean": 127, "ResponseSize_25th_percentile": 2, "ResponseSize_median": 4, "ResponseSize_75th_percentile": 22, "ResponseSize_90th_percentile": 449, "ResponseSize_95th_percentile": 716, "ResponseSize_98th_percentile": 876, "ResponseSize_99th_percentile": 929, "ResponseSize_99.9th_percentile": 977, "ResponseSize_SizeRangeCount_0-10": 15, "ResponseSize_SizeRangeCount_10-100": 3, "ResponseSize_SizeRangeCount_100-1000": 4, "exceptions.UnknownScannerException": 0, "exceptions": 0, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 0, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 55, "RequestSize_min": 80, "RequestSize_max": 340, "RequestSize_mean": 143, "RequestSize_25th_percentile": 97, "RequestSize_median": 116, "RequestSize_75th_percentile": 150, "RequestSize_90th_percentile": 216, "RequestSize_95th_percentile": 280, "RequestSize_98th_percentile": 339, "RequestSize_99th_percentile": 339, "RequestSize_99.9th_percentile": 339, "RequestSize_SizeRangeCount_10-100": 8, "RequestSize_SizeRangeCount_100-1000": 14, "sentBytes": 5103 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:58965", "tag.serverName": "jenkins-hbase20.apache.org,42559,1689121902528", "tag.clusterId": "b94c042f-d510-47b7-9d4c-011dde64d627", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "regionCount": 0, "storeCount": 0, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 0, "maxStoreFileCount": 0, "memStoreSize": 0, "storeFileSize": 0, "maxStoreFileAge": 0, "minStoreFileAge": 3067, "avgStoreFileAge": 3537, "numReferenceFiles": 0, "regionServerStartTime": 1689121902528, "averageRegionSize": 6136, "storeFileIndexSize": 0, "staticIndexSize": 0, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 3948, "localBytesRead": 3948, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803968, "blockCacheCount": 0, "blockCacheDataBlockCount": 0, "blockCacheSize": 601920, "blockCacheCountHitPercent": 0.0, "blockCacheExpressHitPercent": 0.0, "l1CacheSize": 601920, "l1CacheFreeSize": 819803968, "l1CacheCount": 0, "l1CacheEvictionCount": 0, "l1CacheHitCount": 0, "l1CacheMissCount": 0, "l1CacheHitRatio": 0.0, "l1CacheMissRatio": 0.0, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 4608, "ByteBuffAllocatorPoolAllocationBytes": 133120, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 10, "totalRowActionRequestCount": 0, "readRequestCount": 0, "filteredReadRequestCount": 0, "writeRequestCount": 0, "rpcGetRequestCount": 1, "rpcFullScanRequestCount": 0, "rpcScanRequestCount": 1, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 8, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 0, "blockCacheHitCountPrimary": 0, "blockCacheHitCachingCount": 0, "blockCacheMissCount": 0, "blockCacheMissCountPrimary": 0, "blockCacheMissCachingCount": 0, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 0, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 0, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 0, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 0, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 1, "ScanTime_min": 1, "ScanTime_max": 1, "ScanTime_mean": 1, "ScanTime_25th_percentile": 1, "ScanTime_median": 1, "ScanTime_75th_percentile": 1, "ScanTime_90th_percentile": 1, "ScanTime_95th_percentile": 1, "ScanTime_98th_percentile": 1, "ScanTime_99th_percentile": 1, "ScanTime_99.9th_percentile": 1, "ScanTime_TimeRangeCount_0-1": 1, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 8, "Put_min": 2, "Put_max": 2, "Put_mean": 2, "Put_25th_percentile": 2, "Put_median": 2, "Put_75th_percentile": 2, "Put_90th_percentile": 2, "Put_95th_percentile": 2, "Put_98th_percentile": 2, "Put_99th_percentile": 2, "Put_99.9th_percentile": 2, "Put_TimeRangeCount_1-3": 1, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 2, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 1, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 15699, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 2, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 1, "ScanSize_min": 956, "ScanSize_max": 956, "ScanSize_mean": 956, "ScanSize_25th_percentile": 956, "ScanSize_median": 956, "ScanSize_75th_percentile": 956, "ScanSize_90th_percentile": 956, "ScanSize_95th_percentile": 956, "ScanSize_98th_percentile": 956, "ScanSize_99th_percentile": 956, "ScanSize_99.9th_percentile": 956, "ScanSize_SizeRangeCount_100-1000": 1, "slowGetCount": 0, "flushedMemstoreBytes": 1370, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 2, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 28, "ServerReadQueryPerSecond_mean_rate": 1.5853034410975209, "ServerReadQueryPerSecond_1min_rate": 2.4394406617012336, "ServerReadQueryPerSecond_5min_rate": 2.5642928612206086, "ServerReadQueryPerSecond_15min_rate": 2.587885329492682, "ServerWriteQueryPerSecond_count": 22, "ServerWriteQueryPerSecond_mean_rate": 1.2455953139609977, "ServerWriteQueryPerSecond_1min_rate": 1.4159911170741353, "ServerWriteQueryPerSecond_5min_rate": 1.4033057092356764, "ServerWriteQueryPerSecond_15min_rate": 1.4011080303990207 } ] } 2023-07-12 00:32:03,793 INFO [RS:0;jenkins-hbase20:36343] regionserver.HRegionServer(3305): Received CLOSE for a1d134596bd623a40abe345dfeab7ab9 2023-07-12 00:32:03,793 INFO [RS:0;jenkins-hbase20:36343] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,36343,1689121916835 2023-07-12 00:32:03,793 DEBUG [RS:0;jenkins-hbase20:36343] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2afa529e to 127.0.0.1:58965 2023-07-12 00:32:03,794 DEBUG [RS:0;jenkins-hbase20:36343] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:03,794 INFO [RS:0;jenkins-hbase20:36343] regionserver.HRegionServer(1474): Waiting on 2 regions to close 2023-07-12 00:32:03,794 DEBUG [RS:0;jenkins-hbase20:36343] regionserver.HRegionServer(1478): Online Regions={64f62a3e3514274d6ca189fac5775125=testMasterSessionExpired_1689121920970,,1689121923108.64f62a3e3514274d6ca189fac5775125., a1d134596bd623a40abe345dfeab7ab9=hbase:namespace,,1689121917562.a1d134596bd623a40abe345dfeab7ab9.} 2023-07-12 00:32:03,794 DEBUG [RS:0;jenkins-hbase20:36343] regionserver.HRegionServer(1504): Waiting on 64f62a3e3514274d6ca189fac5775125, a1d134596bd623a40abe345dfeab7ab9 2023-07-12 00:32:03,798 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 64f62a3e3514274d6ca189fac5775125, disabling compactions & flushes 2023-07-12 00:32:03,802 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region testMasterSessionExpired_1689121920970,,1689121923108.64f62a3e3514274d6ca189fac5775125. 2023-07-12 00:32:03,799 WARN [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=33137] master.MasterRpcServices(612): jenkins-hbase20.apache.org,36703,1689121916858 reported a fatal error: ***** ABORTING region server jenkins-hbase20.apache.org,36703,1689121916858: killAll ***** 2023-07-12 00:32:03,799 DEBUG [M:0;jenkins-hbase20:33137] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 00:32:03,803 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testMasterSessionExpired_1689121920970,,1689121923108.64f62a3e3514274d6ca189fac5775125. 2023-07-12 00:32:03,804 INFO [M:0;jenkins-hbase20:33137] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:03,804 DEBUG [M:0;jenkins-hbase20:33137] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:03,804 DEBUG [M:0;jenkins-hbase20:33137] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 00:32:03,804 DEBUG [M:0;jenkins-hbase20:33137] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:03,804 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on testMasterSessionExpired_1689121920970,,1689121923108.64f62a3e3514274d6ca189fac5775125. after waiting 0 ms 2023-07-12 00:32:03,806 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region testMasterSessionExpired_1689121920970,,1689121923108.64f62a3e3514274d6ca189fac5775125. 2023-07-12 00:32:03,807 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,36703,1689121916858' ***** 2023-07-12 00:32:03,807 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2311): STOPPED: killAll 2023-07-12 00:32:03,807 INFO [Listener at localhost.localdomain/34481] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-12 00:32:03,809 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50cb0c5c to 127.0.0.1:58965 2023-07-12 00:32:03,809 DEBUG [Listener at localhost.localdomain/34481] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:03,809 DEBUG [Listener at localhost.localdomain/34481] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-12 00:32:03,810 INFO [RS:1;jenkins-hbase20:36703] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 00:32:03,810 INFO [RS:1;jenkins-hbase20:36703] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-12 00:32:03,810 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 00:32:03,810 INFO [RS:1;jenkins-hbase20:36703] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-12 00:32:03,810 INFO [RS:1;jenkins-hbase20:36703] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,36703,1689121916858 2023-07-12 00:32:03,810 DEBUG [RS:1;jenkins-hbase20:36703] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x787c4f9b to 127.0.0.1:58965 2023-07-12 00:32:03,810 DEBUG [RS:1;jenkins-hbase20:36703] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:03,810 INFO [RS:1;jenkins-hbase20:36703] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 00:32:03,810 INFO [RS:1;jenkins-hbase20:36703] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 00:32:03,810 INFO [RS:1;jenkins-hbase20:36703] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 00:32:03,810 INFO [RS:1;jenkins-hbase20:36703] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-12 00:32:03,811 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1825): Memstore data size is 41 in region testMasterSessionExpired_1689121920970,,1689121923108.64f62a3e3514274d6ca189fac5775125. 2023-07-12 00:32:03,812 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed testMasterSessionExpired_1689121920970,,1689121923108.64f62a3e3514274d6ca189fac5775125. 2023-07-12 00:32:03,812 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 64f62a3e3514274d6ca189fac5775125: 2023-07-12 00:32:03,812 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed testMasterSessionExpired_1689121920970,,1689121923108.64f62a3e3514274d6ca189fac5775125. 2023-07-12 00:32:03,814 INFO [RS:1;jenkins-hbase20:36703] regionserver.HRegionServer(1474): Waiting on 1 regions to close 2023-07-12 00:32:03,815 DEBUG [RS:1;jenkins-hbase20:36703] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-07-12 00:32:03,815 DEBUG [RS:1;jenkins-hbase20:36703] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-12 00:32:03,816 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 00:32:03,816 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing a1d134596bd623a40abe345dfeab7ab9, disabling compactions & flushes 2023-07-12 00:32:03,816 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689121917562.a1d134596bd623a40abe345dfeab7ab9. 2023-07-12 00:32:03,816 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689121917562.a1d134596bd623a40abe345dfeab7ab9. 2023-07-12 00:32:03,816 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689121917562.a1d134596bd623a40abe345dfeab7ab9. after waiting 0 ms 2023-07-12 00:32:03,816 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689121917562.a1d134596bd623a40abe345dfeab7ab9. 2023-07-12 00:32:03,816 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 00:32:03,816 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 00:32:03,816 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 00:32:03,816 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 00:32:03,821 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689121917562.a1d134596bd623a40abe345dfeab7ab9. 2023-07-12 00:32:03,822 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689121917562.a1d134596bd623a40abe345dfeab7ab9. 2023-07-12 00:32:03,823 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for a1d134596bd623a40abe345dfeab7ab9: 2023-07-12 00:32:03,823 ERROR [M:0;jenkins-hbase20:33137] regionserver.HRegion(1825): Memstore data size is 14225 in region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:03,823 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689121917562.a1d134596bd623a40abe345dfeab7ab9. 2023-07-12 00:32:03,823 INFO [M:0;jenkins-hbase20:33137] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:03,823 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1825): Memstore data size is 2953 in region hbase:meta,,1.1588230740 2023-07-12 00:32:03,823 DEBUG [M:0;jenkins-hbase20:33137] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 00:32:03,823 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 00:32:03,823 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 00:32:03,823 INFO [M:0;jenkins-hbase20:33137] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 00:32:03,824 INFO [M:0;jenkins-hbase20:33137] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:33137 2023-07-12 00:32:03,824 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 00:32:03,824 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 00:32:03,825 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-12 00:32:03,831 DEBUG [M:0;jenkins-hbase20:33137] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,33137,1689121916771 already deleted, retry=false 2023-07-12 00:32:03,857 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 00:32:03,864 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 00:32:03,941 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:03,941 INFO [M:0;jenkins-hbase20:33137] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,33137,1689121916771; zookeeper connection closed. 2023-07-12 00:32:03,942 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:33137-0x1007e974181000a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:03,994 INFO [RS:0;jenkins-hbase20:36343] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,36343,1689121916835; all regions closed. 2023-07-12 00:32:04,000 DEBUG [RS:0;jenkins-hbase20:36343] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:04,001 INFO [RS:0;jenkins-hbase20:36343] regionserver.LeaseManager(133): Closed leases 2023-07-12 00:32:04,001 INFO [RS:0;jenkins-hbase20:36343] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-12 00:32:04,001 INFO [RS:0;jenkins-hbase20:36343] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 00:32:04,001 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 00:32:04,001 INFO [RS:0;jenkins-hbase20:36343] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 00:32:04,001 INFO [RS:0;jenkins-hbase20:36343] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 00:32:04,002 INFO [RS:0;jenkins-hbase20:36343] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:36343 2023-07-12 00:32:04,005 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:36343-0x1007e974181000c, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,36343,1689121916835 2023-07-12 00:32:04,005 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:36703-0x1007e974181000d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,36343,1689121916835 2023-07-12 00:32:04,005 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:36343-0x1007e974181000c, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:04,005 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:36703-0x1007e974181000d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:04,015 INFO [RS:1;jenkins-hbase20:36703] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,36703,1689121916858; all regions closed. 2023-07-12 00:32:04,023 DEBUG [RS:1;jenkins-hbase20:36703] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:04,023 INFO [RS:1;jenkins-hbase20:36703] regionserver.LeaseManager(133): Closed leases 2023-07-12 00:32:04,024 INFO [RS:1;jenkins-hbase20:36703] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-12 00:32:04,024 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 00:32:04,024 INFO [RS:1;jenkins-hbase20:36703] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:36703 2023-07-12 00:32:04,025 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:36703-0x1007e974181000d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,36703,1689121916858 2023-07-12 00:32:04,345 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:36703-0x1007e974181000d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:04,345 INFO [RS:1;jenkins-hbase20:36703] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,36703,1689121916858; zookeeper connection closed. 2023-07-12 00:32:04,345 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:36703-0x1007e974181000d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:04,345 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@291d5ed6] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@291d5ed6 2023-07-12 00:32:04,445 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:36343-0x1007e974181000c, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:04,445 INFO [RS:0;jenkins-hbase20:36343] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,36343,1689121916835; zookeeper connection closed. 2023-07-12 00:32:04,445 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:36343-0x1007e974181000c, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:04,446 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@7e4741a6] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@7e4741a6 2023-07-12 00:32:04,446 INFO [Listener at localhost.localdomain/34481] util.JVMClusterUtil(335): Shutdown of 2 master(s) and 2 regionserver(s) complete 2023-07-12 00:32:04,547 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810009, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:04,547 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:04,550 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=testing utility connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:32:04,553 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility0x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:32:04,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): testing utility-0x1007e9741810013 connected 2023-07-12 00:32:04,555 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2023-07-12 00:32:04,555 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs 2023-07-12 00:32:04,555 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/splitWAL 2023-07-12 00:32:04,556 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters 2023-07-12 00:32:04,556 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc 2023-07-12 00:32:04,556 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master-maintenance 2023-07-12 00:32:04,557 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot 2023-07-12 00:32:04,557 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/switch 2023-07-12 00:32:04,557 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-12 00:32:04,558 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/draining 2023-07-12 00:32:04,558 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-07-12 00:32:04,558 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/hbaseid 2023-07-12 00:32:04,559 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/table 2023-07-12 00:32:04,560 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/abort 2023-07-12 00:32:04,561 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired 2023-07-12 00:32:04,561 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached 2023-07-12 00:32:04,562 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/abort 2023-07-12 00:32:04,562 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired 2023-07-12 00:32:04,562 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached 2023-07-12 00:32:04,563 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/default 2023-07-12 00:32:04,564 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/hbase 2023-07-12 00:32:04,564 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/table/hbase:meta 2023-07-12 00:32:04,565 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/table/hbase:namespace 2023-07-12 00:32:04,565 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/table/testMasterSessionExpired_1689121920970 2023-07-12 00:32:04,569 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table/testMasterSessionExpired_1689121920970 2023-07-12 00:32:04,569 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/table 2023-07-12 00:32:04,569 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table/hbase:namespace 2023-07-12 00:32:04,569 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table/hbase:meta 2023-07-12 00:32:04,570 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 00:32:04,570 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 00:32:04,570 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 00:32:04,570 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached 2023-07-12 00:32:04,570 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot 2023-07-12 00:32:04,570 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 00:32:04,570 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 00:32:04,570 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached 2023-07-12 00:32:04,570 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc 2023-07-12 00:32:04,570 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 00:32:04,570 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 00:32:04,570 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table 2023-07-12 00:32:04,570 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:04,570 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/hbaseid 2023-07-12 00:32:04,570 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace 2023-07-12 00:32:04,570 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/draining 2023-07-12 00:32:04,571 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 00:32:04,571 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/switch 2023-07-12 00:32:04,571 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot 2023-07-12 00:32:04,571 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master-maintenance 2023-07-12 00:32:04,571 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc 2023-07-12 00:32:04,571 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:32:04,571 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL 2023-07-12 00:32:04,571 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:04,571 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 00:32:04,571 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase 2023-07-12 00:32:04,586 INFO [Listener at localhost.localdomain/34481] hbase.ResourceChecker(175): after: TestZooKeeper#testMasterSessionExpired Thread=293 (was 263) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/cluster_2bc97589-c177-c63b-7f92-2672baa4b2fe/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/cluster_2bc97589-c177-c63b-7f92-2672baa4b2fe/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost.localdomain:44295 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1879248547) connection to localhost.localdomain/127.0.0.1:44295 from jenkins.hfs.2 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: IPC Client (1879248547) connection to localhost.localdomain/127.0.0.1:44295 from jenkins.hfs.3 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-7-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1628992759_17 at /127.0.0.1:55972 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1601477482_17 at /127.0.0.1:47422 [Waiting for operation #8] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.3@localhost.localdomain:44295 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=644 (was 610) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=176 (was 182), ProcessCount=172 (was 172), AvailableMemoryMB=5661 (was 4655) - AvailableMemoryMB LEAK? - 2023-07-12 00:32:04,598 INFO [Listener at localhost.localdomain/34481] hbase.ResourceChecker(147): before: TestZooKeeper#testMasterZKSessionRecoveryFailure Thread=293, OpenFileDescriptor=644, MaxFileDescriptor=60000, SystemLoadAverage=176, ProcessCount=172, AvailableMemoryMB=5658 2023-07-12 00:32:04,611 INFO [Listener at localhost.localdomain/34481] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983 with version=8 2023-07-12 00:32:04,611 INFO [Listener at localhost.localdomain/34481] hbase.HBaseTestingUtility(1408): The hbase.fs.tmp.dir is set to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/hbase-staging 2023-07-12 00:32:04,613 INFO [Listener at localhost.localdomain/34481] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-07-12 00:32:04,613 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:04,614 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:04,614 INFO [Listener at localhost.localdomain/34481] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 00:32:04,614 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:04,614 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 00:32:04,614 INFO [Listener at localhost.localdomain/34481] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 00:32:04,615 INFO [Listener at localhost.localdomain/34481] ipc.NettyRpcServer(120): Bind to /148.251.75.209:45201 2023-07-12 00:32:04,615 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:04,616 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:04,617 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=master:45201 connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:32:04,620 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:452010x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:32:04,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:45201-0x1007e9741810014 connected 2023-07-12 00:32:04,627 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:04,628 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 00:32:04,628 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 00:32:04,630 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45201 2023-07-12 00:32:04,630 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45201 2023-07-12 00:32:04,630 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45201 2023-07-12 00:32:04,630 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45201 2023-07-12 00:32:04,630 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45201 2023-07-12 00:32:04,630 INFO [Listener at localhost.localdomain/34481] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983, hbase.cluster.distributed=false 2023-07-12 00:32:04,635 INFO [Listener at localhost.localdomain/34481] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-07-12 00:32:04,635 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:04,635 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:04,635 INFO [Listener at localhost.localdomain/34481] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 00:32:04,635 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:04,635 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 00:32:04,635 INFO [Listener at localhost.localdomain/34481] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 00:32:04,636 INFO [Listener at localhost.localdomain/34481] ipc.NettyRpcServer(120): Bind to /148.251.75.209:46459 2023-07-12 00:32:04,636 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:04,637 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:04,638 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=master:46459 connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:32:04,642 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:464590x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:32:04,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:46459-0x1007e9741810015 connected 2023-07-12 00:32:04,644 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:04,644 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 00:32:04,645 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 00:32:04,646 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46459 2023-07-12 00:32:04,650 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46459 2023-07-12 00:32:04,651 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46459 2023-07-12 00:32:04,651 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46459 2023-07-12 00:32:04,653 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46459 2023-07-12 00:32:04,653 INFO [Listener at localhost.localdomain/34481] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983, hbase.cluster.distributed=false 2023-07-12 00:32:04,666 INFO [Listener at localhost.localdomain/34481] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-07-12 00:32:04,666 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:04,666 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:04,666 INFO [Listener at localhost.localdomain/34481] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 00:32:04,666 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:04,666 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 00:32:04,666 INFO [Listener at localhost.localdomain/34481] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 00:32:04,671 INFO [Listener at localhost.localdomain/34481] ipc.NettyRpcServer(120): Bind to /148.251.75.209:34207 2023-07-12 00:32:04,671 INFO [Listener at localhost.localdomain/34481] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 00:32:04,672 DEBUG [Listener at localhost.localdomain/34481] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 00:32:04,673 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:04,673 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:04,674 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34207 connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:32:04,678 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:342070x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:32:04,679 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:342070x0, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:04,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:34207-0x1007e9741810016 connected 2023-07-12 00:32:04,680 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:34207-0x1007e9741810016, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 00:32:04,681 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:34207-0x1007e9741810016, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 00:32:04,683 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34207 2023-07-12 00:32:04,684 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34207 2023-07-12 00:32:04,684 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34207 2023-07-12 00:32:04,685 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34207 2023-07-12 00:32:04,686 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34207 2023-07-12 00:32:04,697 INFO [Listener at localhost.localdomain/34481] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-07-12 00:32:04,697 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:04,698 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:04,698 INFO [Listener at localhost.localdomain/34481] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 00:32:04,698 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:04,698 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 00:32:04,698 INFO [Listener at localhost.localdomain/34481] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 00:32:04,699 INFO [Listener at localhost.localdomain/34481] ipc.NettyRpcServer(120): Bind to /148.251.75.209:43787 2023-07-12 00:32:04,699 INFO [Listener at localhost.localdomain/34481] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 00:32:04,701 DEBUG [Listener at localhost.localdomain/34481] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 00:32:04,701 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:04,702 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:04,703 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:43787 connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:32:04,706 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:437870x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:32:04,708 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:437870x0, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:04,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:43787-0x1007e9741810017 connected 2023-07-12 00:32:04,709 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:43787-0x1007e9741810017, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 00:32:04,709 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:43787-0x1007e9741810017, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 00:32:04,714 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43787 2023-07-12 00:32:04,714 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43787 2023-07-12 00:32:04,714 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43787 2023-07-12 00:32:04,715 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43787 2023-07-12 00:32:04,715 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43787 2023-07-12 00:32:04,717 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,45201,1689121924613 2023-07-12 00:32:04,717 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,46459,1689121924634 2023-07-12 00:32:04,722 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:32:04,722 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:32:04,722 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,45201,1689121924613 2023-07-12 00:32:04,722 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,46459,1689121924634 2023-07-12 00:32:04,727 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:43787-0x1007e9741810017, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:04,727 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:04,728 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:04,727 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:34207-0x1007e9741810016, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:04,727 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:04,728 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:04,730 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:32:04,730 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:32:04,731 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,45201,1689121924613 from backup master directory 2023-07-12 00:32:04,731 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(278): Another master is the active master, jenkins-hbase20.apache.org,45201,1689121924613; waiting to become the next active master 2023-07-12 00:32:04,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:32:04,735 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:32:04,736 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 00:32:04,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:32:04,735 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,45201,1689121924613 2023-07-12 00:32:04,736 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,45201,1689121924613 2023-07-12 00:32:04,736 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:32:04,755 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/hbase.id with ID: da9302d2-b6bf-4aef-9465-3f2de7f4b5fa 2023-07-12 00:32:04,769 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:04,772 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:04,772 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:04,791 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x1e8961d1 to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:32:04,801 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@290281c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:32:04,801 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 00:32:04,802 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 00:32:04,802 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:32:04,804 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store-tmp 2023-07-12 00:32:04,826 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:04,826 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 00:32:04,826 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:04,826 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:04,826 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 00:32:04,826 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:04,826 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:04,826 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 00:32:04,827 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,45201,1689121924613 2023-07-12 00:32:04,831 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C45201%2C1689121924613, suffix=, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,45201,1689121924613, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/oldWALs, maxLogs=10 2023-07-12 00:32:04,851 DEBUG [RS-EventLoopGroup-14-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:32:04,851 DEBUG [RS-EventLoopGroup-14-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:32:04,859 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,45201,1689121924613/jenkins-hbase20.apache.org%2C45201%2C1689121924613.1689121924832 2023-07-12 00:32:04,859 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK], DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK]] 2023-07-12 00:32:04,859 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 00:32:04,859 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:04,859 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:04,859 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:04,862 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:04,864 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 00:32:04,864 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 00:32:04,865 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:04,865 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:04,866 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:04,874 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:04,877 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:04,878 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65699080, jitterRate=-0.021007418632507324}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:04,878 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 00:32:04,878 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 00:32:04,880 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-12 00:32:04,880 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-12 00:32:04,881 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 00:32:04,881 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 00:32:04,882 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-07-12 00:32:04,882 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 00:32:04,887 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-12 00:32:04,889 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-12 00:32:04,898 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 00:32:04,899 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 00:32:04,899 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 00:32:04,899 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 00:32:04,903 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:04,903 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:04,903 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 00:32:04,904 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 00:32:04,904 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 00:32:04,905 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:34207-0x1007e9741810016, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 00:32:04,905 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 00:32:04,905 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:43787-0x1007e9741810017, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 00:32:04,905 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:04,905 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 00:32:04,906 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:04,906 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,45201,1689121924613, sessionid=0x1007e9741810014, setting cluster-up flag (Was=false) 2023-07-12 00:32:04,911 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:04,911 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:04,911 INFO [M:1;jenkins-hbase20:46459] regionserver.HRegionServer(951): ClusterId : da9302d2-b6bf-4aef-9465-3f2de7f4b5fa 2023-07-12 00:32:04,913 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 00:32:04,916 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,45201,1689121924613 2023-07-12 00:32:04,932 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:04,932 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:04,942 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 00:32:04,944 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,45201,1689121924613 2023-07-12 00:32:04,945 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.hbase-snapshot/.tmp 2023-07-12 00:32:04,953 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-12 00:32:04,954 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:32:04,954 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:32:04,954 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:32:04,954 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:32:04,954 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-07-12 00:32:04,954 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:04,954 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-12 00:32:04,954 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:04,975 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689121954975 2023-07-12 00:32:04,975 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 00:32:04,975 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 00:32:04,976 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 00:32:04,976 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 00:32:04,976 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 00:32:04,976 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 00:32:04,976 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 00:32:04,976 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-12 00:32:04,977 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:04,978 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 00:32:04,978 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 00:32:04,978 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 00:32:04,979 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 00:32:04,979 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 00:32:04,979 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 00:32:04,981 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689121924979,5,FailOnTimeoutGroup] 2023-07-12 00:32:04,990 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689121924982,5,FailOnTimeoutGroup] 2023-07-12 00:32:04,990 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:04,990 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 00:32:04,990 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:04,990 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:05,023 INFO [RS:0;jenkins-hbase20:34207] regionserver.HRegionServer(951): ClusterId : da9302d2-b6bf-4aef-9465-3f2de7f4b5fa 2023-07-12 00:32:05,028 INFO [RS:1;jenkins-hbase20:43787] regionserver.HRegionServer(951): ClusterId : da9302d2-b6bf-4aef-9465-3f2de7f4b5fa 2023-07-12 00:32:05,028 DEBUG [RS:0;jenkins-hbase20:34207] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 00:32:05,031 DEBUG [RS:1;jenkins-hbase20:43787] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 00:32:05,035 DEBUG [RS:0;jenkins-hbase20:34207] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 00:32:05,035 DEBUG [RS:0;jenkins-hbase20:34207] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 00:32:05,035 DEBUG [RS:1;jenkins-hbase20:43787] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 00:32:05,035 DEBUG [RS:1;jenkins-hbase20:43787] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 00:32:05,036 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 00:32:05,037 DEBUG [RS:0;jenkins-hbase20:34207] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 00:32:05,040 DEBUG [RS:0;jenkins-hbase20:34207] zookeeper.ReadOnlyZKClient(139): Connect 0x615b8eac to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:32:05,040 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 00:32:05,041 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983 2023-07-12 00:32:05,042 DEBUG [RS:1;jenkins-hbase20:43787] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 00:32:05,052 DEBUG [RS:1;jenkins-hbase20:43787] zookeeper.ReadOnlyZKClient(139): Connect 0x27f7c753 to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:32:05,068 DEBUG [RS:0;jenkins-hbase20:34207] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f9c7cf7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:32:05,069 DEBUG [RS:0;jenkins-hbase20:34207] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ba3e5dc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-12 00:32:05,079 DEBUG [RS:1;jenkins-hbase20:43787] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29302ddd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:32:05,080 DEBUG [RS:1;jenkins-hbase20:43787] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d0fac23, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-12 00:32:05,085 DEBUG [RS:0;jenkins-hbase20:34207] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:34207 2023-07-12 00:32:05,085 INFO [RS:0;jenkins-hbase20:34207] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 00:32:05,085 INFO [RS:0;jenkins-hbase20:34207] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 00:32:05,085 DEBUG [RS:0;jenkins-hbase20:34207] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 00:32:05,086 INFO [RS:0;jenkins-hbase20:34207] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase20.apache.org,45201,1689121924613 with isa=jenkins-hbase20.apache.org/148.251.75.209:34207, startcode=1689121924665 2023-07-12 00:32:05,086 DEBUG [RS:0;jenkins-hbase20:34207] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 00:32:05,088 INFO [RS-EventLoopGroup-11-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:58627, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.4 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 00:32:05,091 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=45201] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,34207,1689121924665 2023-07-12 00:32:05,091 DEBUG [RS:0;jenkins-hbase20:34207] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983 2023-07-12 00:32:05,092 DEBUG [RS:0;jenkins-hbase20:34207] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:44295 2023-07-12 00:32:05,092 DEBUG [RS:0;jenkins-hbase20:34207] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 00:32:05,093 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:05,094 DEBUG [RS:1;jenkins-hbase20:43787] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase20:43787 2023-07-12 00:32:05,094 INFO [RS:1;jenkins-hbase20:43787] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 00:32:05,094 INFO [RS:1;jenkins-hbase20:43787] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 00:32:05,094 DEBUG [RS:0;jenkins-hbase20:34207] zookeeper.ZKUtil(162): regionserver:34207-0x1007e9741810016, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,34207,1689121924665 2023-07-12 00:32:05,094 DEBUG [RS:1;jenkins-hbase20:43787] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 00:32:05,094 WARN [RS:0;jenkins-hbase20:34207] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 00:32:05,094 INFO [RS:0;jenkins-hbase20:34207] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:32:05,094 DEBUG [RS:0;jenkins-hbase20:34207] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,34207,1689121924665 2023-07-12 00:32:05,094 INFO [RS:1;jenkins-hbase20:43787] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase20.apache.org,45201,1689121924613 with isa=jenkins-hbase20.apache.org/148.251.75.209:43787, startcode=1689121924697 2023-07-12 00:32:05,095 DEBUG [RS:1;jenkins-hbase20:43787] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 00:32:05,096 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,34207,1689121924665] 2023-07-12 00:32:05,103 INFO [RS-EventLoopGroup-11-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57335, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.5 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 00:32:05,103 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=45201] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,43787,1689121924697 2023-07-12 00:32:05,104 DEBUG [RS:1;jenkins-hbase20:43787] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983 2023-07-12 00:32:05,104 DEBUG [RS:1;jenkins-hbase20:43787] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:44295 2023-07-12 00:32:05,104 DEBUG [RS:1;jenkins-hbase20:43787] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 00:32:05,104 DEBUG [RS:0;jenkins-hbase20:34207] zookeeper.ZKUtil(162): regionserver:34207-0x1007e9741810016, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,34207,1689121924665 2023-07-12 00:32:05,105 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:34207-0x1007e9741810016, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:05,105 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:05,105 DEBUG [RS:1;jenkins-hbase20:43787] zookeeper.ZKUtil(162): regionserver:43787-0x1007e9741810017, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,43787,1689121924697 2023-07-12 00:32:05,106 WARN [RS:1;jenkins-hbase20:43787] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 00:32:05,106 INFO [RS:1;jenkins-hbase20:43787] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:32:05,106 DEBUG [RS:1;jenkins-hbase20:43787] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,43787,1689121924697 2023-07-12 00:32:05,108 DEBUG [RS:0;jenkins-hbase20:34207] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 00:32:05,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34207-0x1007e9741810016, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,34207,1689121924665 2023-07-12 00:32:05,108 INFO [RS:0;jenkins-hbase20:34207] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 00:32:05,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34207-0x1007e9741810016, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,43787,1689121924697 2023-07-12 00:32:05,109 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,43787,1689121924697] 2023-07-12 00:32:05,113 INFO [RS:0;jenkins-hbase20:34207] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 00:32:05,114 INFO [RS:0;jenkins-hbase20:34207] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 00:32:05,114 INFO [RS:0;jenkins-hbase20:34207] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:05,114 INFO [RS:0;jenkins-hbase20:34207] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 00:32:05,115 DEBUG [RS:1;jenkins-hbase20:43787] zookeeper.ZKUtil(162): regionserver:43787-0x1007e9741810017, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,34207,1689121924665 2023-07-12 00:32:05,116 INFO [RS:0;jenkins-hbase20:34207] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:05,116 DEBUG [RS:1;jenkins-hbase20:43787] zookeeper.ZKUtil(162): regionserver:43787-0x1007e9741810017, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,43787,1689121924697 2023-07-12 00:32:05,117 DEBUG [RS:0;jenkins-hbase20:34207] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:05,117 DEBUG [RS:0;jenkins-hbase20:34207] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:05,117 DEBUG [RS:0;jenkins-hbase20:34207] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:05,117 DEBUG [RS:0;jenkins-hbase20:34207] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:05,117 DEBUG [RS:0;jenkins-hbase20:34207] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:05,117 DEBUG [RS:0;jenkins-hbase20:34207] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-12 00:32:05,117 DEBUG [RS:0;jenkins-hbase20:34207] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:05,117 DEBUG [RS:0;jenkins-hbase20:34207] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:05,117 DEBUG [RS:1;jenkins-hbase20:43787] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 00:32:05,117 DEBUG [RS:0;jenkins-hbase20:34207] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:05,118 DEBUG [RS:0;jenkins-hbase20:34207] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:05,118 INFO [RS:1;jenkins-hbase20:43787] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 00:32:05,122 INFO [RS:0;jenkins-hbase20:34207] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:05,123 INFO [RS:0;jenkins-hbase20:34207] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:05,123 INFO [RS:1;jenkins-hbase20:43787] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 00:32:05,123 INFO [RS:0;jenkins-hbase20:34207] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:05,124 INFO [RS:1;jenkins-hbase20:43787] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 00:32:05,124 INFO [RS:1;jenkins-hbase20:43787] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:05,126 INFO [RS:1;jenkins-hbase20:43787] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 00:32:05,128 INFO [RS:1;jenkins-hbase20:43787] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:05,128 DEBUG [RS:1;jenkins-hbase20:43787] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:05,128 DEBUG [RS:1;jenkins-hbase20:43787] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:05,128 DEBUG [RS:1;jenkins-hbase20:43787] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:05,128 DEBUG [RS:1;jenkins-hbase20:43787] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:05,128 DEBUG [RS:1;jenkins-hbase20:43787] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:05,128 DEBUG [RS:1;jenkins-hbase20:43787] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-12 00:32:05,128 DEBUG [RS:1;jenkins-hbase20:43787] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:05,128 DEBUG [RS:1;jenkins-hbase20:43787] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:05,129 DEBUG [RS:1;jenkins-hbase20:43787] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:05,129 DEBUG [RS:1;jenkins-hbase20:43787] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:05,135 INFO [RS:1;jenkins-hbase20:43787] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:05,135 INFO [RS:1;jenkins-hbase20:43787] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:05,135 INFO [RS:1;jenkins-hbase20:43787] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:05,141 INFO [RS:0;jenkins-hbase20:34207] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 00:32:05,141 INFO [RS:0;jenkins-hbase20:34207] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34207,1689121924665-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:05,149 INFO [RS:1;jenkins-hbase20:43787] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 00:32:05,149 INFO [RS:1;jenkins-hbase20:43787] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,43787,1689121924697-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:05,157 INFO [RS:0;jenkins-hbase20:34207] regionserver.Replication(203): jenkins-hbase20.apache.org,34207,1689121924665 started 2023-07-12 00:32:05,157 INFO [RS:0;jenkins-hbase20:34207] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,34207,1689121924665, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:34207, sessionid=0x1007e9741810016 2023-07-12 00:32:05,157 DEBUG [RS:0;jenkins-hbase20:34207] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 00:32:05,157 DEBUG [RS:0;jenkins-hbase20:34207] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,34207,1689121924665 2023-07-12 00:32:05,157 DEBUG [RS:0;jenkins-hbase20:34207] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,34207,1689121924665' 2023-07-12 00:32:05,157 DEBUG [RS:0;jenkins-hbase20:34207] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 00:32:05,158 DEBUG [RS:0;jenkins-hbase20:34207] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 00:32:05,158 DEBUG [RS:0;jenkins-hbase20:34207] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 00:32:05,158 DEBUG [RS:0;jenkins-hbase20:34207] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 00:32:05,158 DEBUG [RS:0;jenkins-hbase20:34207] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,34207,1689121924665 2023-07-12 00:32:05,158 DEBUG [RS:0;jenkins-hbase20:34207] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,34207,1689121924665' 2023-07-12 00:32:05,158 DEBUG [RS:0;jenkins-hbase20:34207] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 00:32:05,159 DEBUG [RS:0;jenkins-hbase20:34207] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 00:32:05,159 DEBUG [RS:0;jenkins-hbase20:34207] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 00:32:05,159 INFO [RS:0;jenkins-hbase20:34207] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 00:32:05,159 INFO [RS:0;jenkins-hbase20:34207] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 00:32:05,162 INFO [RS:1;jenkins-hbase20:43787] regionserver.Replication(203): jenkins-hbase20.apache.org,43787,1689121924697 started 2023-07-12 00:32:05,162 INFO [RS:1;jenkins-hbase20:43787] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,43787,1689121924697, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:43787, sessionid=0x1007e9741810017 2023-07-12 00:32:05,162 DEBUG [RS:1;jenkins-hbase20:43787] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 00:32:05,162 DEBUG [RS:1;jenkins-hbase20:43787] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,43787,1689121924697 2023-07-12 00:32:05,162 DEBUG [RS:1;jenkins-hbase20:43787] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,43787,1689121924697' 2023-07-12 00:32:05,162 DEBUG [RS:1;jenkins-hbase20:43787] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 00:32:05,163 DEBUG [RS:1;jenkins-hbase20:43787] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 00:32:05,163 DEBUG [RS:1;jenkins-hbase20:43787] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 00:32:05,164 DEBUG [RS:1;jenkins-hbase20:43787] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 00:32:05,164 DEBUG [RS:1;jenkins-hbase20:43787] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,43787,1689121924697 2023-07-12 00:32:05,164 DEBUG [RS:1;jenkins-hbase20:43787] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,43787,1689121924697' 2023-07-12 00:32:05,164 DEBUG [RS:1;jenkins-hbase20:43787] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 00:32:05,164 DEBUG [RS:1;jenkins-hbase20:43787] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 00:32:05,164 DEBUG [RS:1;jenkins-hbase20:43787] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 00:32:05,164 INFO [RS:1;jenkins-hbase20:43787] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 00:32:05,164 INFO [RS:1;jenkins-hbase20:43787] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 00:32:05,261 INFO [RS:0;jenkins-hbase20:34207] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C34207%2C1689121924665, suffix=, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,34207,1689121924665, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/oldWALs, maxLogs=32 2023-07-12 00:32:05,266 INFO [RS:1;jenkins-hbase20:43787] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C43787%2C1689121924697, suffix=, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,43787,1689121924697, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/oldWALs, maxLogs=32 2023-07-12 00:32:05,284 DEBUG [RS-EventLoopGroup-14-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:32:05,284 DEBUG [RS-EventLoopGroup-14-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:32:05,291 INFO [RS:0;jenkins-hbase20:34207] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,34207,1689121924665/jenkins-hbase20.apache.org%2C34207%2C1689121924665.1689121925262 2023-07-12 00:32:05,293 DEBUG [RS:0;jenkins-hbase20:34207] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK], DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK]] 2023-07-12 00:32:05,293 DEBUG [RS-EventLoopGroup-14-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:32:05,293 DEBUG [RS-EventLoopGroup-14-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:32:05,299 INFO [RS:1;jenkins-hbase20:43787] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,43787,1689121924697/jenkins-hbase20.apache.org%2C43787%2C1689121924697.1689121925267 2023-07-12 00:32:05,299 DEBUG [RS:1;jenkins-hbase20:43787] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK], DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK]] 2023-07-12 00:32:05,480 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:05,481 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 00:32:05,483 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/info 2023-07-12 00:32:05,484 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 00:32:05,484 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:05,484 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 00:32:05,486 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/rep_barrier 2023-07-12 00:32:05,487 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 00:32:05,487 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:05,487 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 00:32:05,489 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/table 2023-07-12 00:32:05,489 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 00:32:05,490 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:05,491 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740 2023-07-12 00:32:05,491 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740 2023-07-12 00:32:05,493 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 00:32:05,494 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 00:32:05,496 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:05,497 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74659542, jitterRate=0.11251387000083923}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 00:32:05,497 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 00:32:05,497 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 00:32:05,497 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 00:32:05,497 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 00:32:05,497 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 00:32:05,497 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 00:32:05,498 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 00:32:05,498 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 00:32:05,499 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 00:32:05,499 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-12 00:32:05,499 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-12 00:32:05,501 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-12 00:32:05,503 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-12 00:32:05,653 DEBUG [jenkins-hbase20:45201] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-12 00:32:05,654 DEBUG [jenkins-hbase20:45201] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-07-12 00:32:05,656 DEBUG [jenkins-hbase20:45201] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 00:32:05,656 DEBUG [jenkins-hbase20:45201] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 00:32:05,656 DEBUG [jenkins-hbase20:45201] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 00:32:05,657 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,43787,1689121924697, state=OPENING 2023-07-12 00:32:05,658 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-12 00:32:05,659 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:05,659 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:05,661 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 00:32:05,661 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 00:32:05,661 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,43787,1689121924697}] 2023-07-12 00:32:05,817 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,43787,1689121924697 2023-07-12 00:32:05,817 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 00:32:05,819 INFO [RS-EventLoopGroup-14-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34446, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 00:32:05,823 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-12 00:32:05,823 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:32:05,825 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C43787%2C1689121924697.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,43787,1689121924697, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/oldWALs, maxLogs=32 2023-07-12 00:32:05,860 DEBUG [RS-EventLoopGroup-14-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:32:05,860 DEBUG [RS-EventLoopGroup-14-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:32:05,866 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,43787,1689121924697/jenkins-hbase20.apache.org%2C43787%2C1689121924697.meta.1689121925826.meta 2023-07-12 00:32:05,869 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK], DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK]] 2023-07-12 00:32:05,869 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-12 00:32:05,869 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-12 00:32:05,869 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-12 00:32:05,869 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-12 00:32:05,869 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-12 00:32:05,870 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:05,870 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-12 00:32:05,870 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-12 00:32:05,872 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 00:32:05,873 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/info 2023-07-12 00:32:05,874 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/info 2023-07-12 00:32:05,874 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 00:32:05,875 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:05,875 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 00:32:05,877 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/rep_barrier 2023-07-12 00:32:05,877 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/rep_barrier 2023-07-12 00:32:05,877 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 00:32:05,878 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:05,878 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 00:32:05,879 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/table 2023-07-12 00:32:05,879 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/table 2023-07-12 00:32:05,880 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 00:32:05,880 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:05,881 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740 2023-07-12 00:32:05,882 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740 2023-07-12 00:32:05,885 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 00:32:05,887 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 00:32:05,888 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65422656, jitterRate=-0.02512645721435547}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 00:32:05,888 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 00:32:05,889 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689121925817 2023-07-12 00:32:05,894 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-12 00:32:05,895 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-12 00:32:05,896 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,43787,1689121924697, state=OPEN 2023-07-12 00:32:05,897 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 00:32:05,897 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 00:32:05,897 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 00:32:05,897 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 00:32:05,899 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-12 00:32:05,900 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,43787,1689121924697 in 236 msec 2023-07-12 00:32:05,902 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-12 00:32:05,902 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 401 msec 2023-07-12 00:32:05,905 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 952 msec 2023-07-12 00:32:05,905 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689121925905, completionTime=-1 2023-07-12 00:32:05,905 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-12 00:32:05,905 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 00:32:05,908 DEBUG [hconnection-0x223c6d69-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 00:32:05,911 INFO [RS-EventLoopGroup-14-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34460, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 00:32:05,912 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-12 00:32:05,912 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689121985912 2023-07-12 00:32:05,913 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689122045913 2023-07-12 00:32:05,913 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 7 msec 2023-07-12 00:32:05,919 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,45201,1689121924613-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:05,920 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,45201,1689121924613-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:05,920 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,45201,1689121924613-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:05,920 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:45201, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:05,920 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:05,920 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-12 00:32:05,920 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 00:32:05,921 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-12 00:32:05,921 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 00:32:05,923 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 00:32:05,925 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 00:32:05,927 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/hbase/namespace/3d92e4b6acd18630a20a2cb252f2c7f6 2023-07-12 00:32:05,927 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/hbase/namespace/3d92e4b6acd18630a20a2cb252f2c7f6 empty. 2023-07-12 00:32:05,928 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/hbase/namespace/3d92e4b6acd18630a20a2cb252f2c7f6 2023-07-12 00:32:05,928 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-12 00:32:05,945 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-12 00:32:05,947 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 3d92e4b6acd18630a20a2cb252f2c7f6, NAME => 'hbase:namespace,,1689121925920.3d92e4b6acd18630a20a2cb252f2c7f6.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp 2023-07-12 00:32:05,961 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689121925920.3d92e4b6acd18630a20a2cb252f2c7f6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:05,961 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 3d92e4b6acd18630a20a2cb252f2c7f6, disabling compactions & flushes 2023-07-12 00:32:05,961 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689121925920.3d92e4b6acd18630a20a2cb252f2c7f6. 2023-07-12 00:32:05,961 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689121925920.3d92e4b6acd18630a20a2cb252f2c7f6. 2023-07-12 00:32:05,961 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689121925920.3d92e4b6acd18630a20a2cb252f2c7f6. after waiting 0 ms 2023-07-12 00:32:05,961 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689121925920.3d92e4b6acd18630a20a2cb252f2c7f6. 2023-07-12 00:32:05,961 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689121925920.3d92e4b6acd18630a20a2cb252f2c7f6. 2023-07-12 00:32:05,961 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 3d92e4b6acd18630a20a2cb252f2c7f6: 2023-07-12 00:32:05,964 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 00:32:05,965 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689121925920.3d92e4b6acd18630a20a2cb252f2c7f6.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689121925965"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689121925965"}]},"ts":"1689121925965"} 2023-07-12 00:32:05,968 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 00:32:05,969 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 00:32:05,970 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689121925969"}]},"ts":"1689121925969"} 2023-07-12 00:32:05,972 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-12 00:32:05,974 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-07-12 00:32:05,975 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 00:32:05,975 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 00:32:05,975 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 00:32:05,975 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=3d92e4b6acd18630a20a2cb252f2c7f6, ASSIGN}] 2023-07-12 00:32:05,979 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=3d92e4b6acd18630a20a2cb252f2c7f6, ASSIGN 2023-07-12 00:32:05,981 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=3d92e4b6acd18630a20a2cb252f2c7f6, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,43787,1689121924697; forceNewPlan=false, retain=false 2023-07-12 00:32:06,131 INFO [jenkins-hbase20:45201] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-12 00:32:06,133 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=3d92e4b6acd18630a20a2cb252f2c7f6, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,43787,1689121924697 2023-07-12 00:32:06,133 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689121925920.3d92e4b6acd18630a20a2cb252f2c7f6.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689121926132"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121926132"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121926132"}]},"ts":"1689121926132"} 2023-07-12 00:32:06,135 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 3d92e4b6acd18630a20a2cb252f2c7f6, server=jenkins-hbase20.apache.org,43787,1689121924697}] 2023-07-12 00:32:06,292 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689121925920.3d92e4b6acd18630a20a2cb252f2c7f6. 2023-07-12 00:32:06,292 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3d92e4b6acd18630a20a2cb252f2c7f6, NAME => 'hbase:namespace,,1689121925920.3d92e4b6acd18630a20a2cb252f2c7f6.', STARTKEY => '', ENDKEY => ''} 2023-07-12 00:32:06,292 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 3d92e4b6acd18630a20a2cb252f2c7f6 2023-07-12 00:32:06,293 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689121925920.3d92e4b6acd18630a20a2cb252f2c7f6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:06,293 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 3d92e4b6acd18630a20a2cb252f2c7f6 2023-07-12 00:32:06,293 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 3d92e4b6acd18630a20a2cb252f2c7f6 2023-07-12 00:32:06,294 INFO [StoreOpener-3d92e4b6acd18630a20a2cb252f2c7f6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 3d92e4b6acd18630a20a2cb252f2c7f6 2023-07-12 00:32:06,296 DEBUG [StoreOpener-3d92e4b6acd18630a20a2cb252f2c7f6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/3d92e4b6acd18630a20a2cb252f2c7f6/info 2023-07-12 00:32:06,296 DEBUG [StoreOpener-3d92e4b6acd18630a20a2cb252f2c7f6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/3d92e4b6acd18630a20a2cb252f2c7f6/info 2023-07-12 00:32:06,296 INFO [StoreOpener-3d92e4b6acd18630a20a2cb252f2c7f6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3d92e4b6acd18630a20a2cb252f2c7f6 columnFamilyName info 2023-07-12 00:32:06,297 INFO [StoreOpener-3d92e4b6acd18630a20a2cb252f2c7f6-1] regionserver.HStore(310): Store=3d92e4b6acd18630a20a2cb252f2c7f6/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:06,298 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/3d92e4b6acd18630a20a2cb252f2c7f6 2023-07-12 00:32:06,298 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/3d92e4b6acd18630a20a2cb252f2c7f6 2023-07-12 00:32:06,302 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 3d92e4b6acd18630a20a2cb252f2c7f6 2023-07-12 00:32:06,305 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/3d92e4b6acd18630a20a2cb252f2c7f6/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:06,306 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 3d92e4b6acd18630a20a2cb252f2c7f6; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71597000, jitterRate=0.06687843799591064}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:06,306 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 3d92e4b6acd18630a20a2cb252f2c7f6: 2023-07-12 00:32:06,307 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689121925920.3d92e4b6acd18630a20a2cb252f2c7f6., pid=6, masterSystemTime=1689121926288 2023-07-12 00:32:06,310 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689121925920.3d92e4b6acd18630a20a2cb252f2c7f6. 2023-07-12 00:32:06,310 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689121925920.3d92e4b6acd18630a20a2cb252f2c7f6. 2023-07-12 00:32:06,310 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=3d92e4b6acd18630a20a2cb252f2c7f6, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,43787,1689121924697 2023-07-12 00:32:06,311 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689121925920.3d92e4b6acd18630a20a2cb252f2c7f6.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689121926310"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121926310"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121926310"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121926310"}]},"ts":"1689121926310"} 2023-07-12 00:32:06,316 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-12 00:32:06,316 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 3d92e4b6acd18630a20a2cb252f2c7f6, server=jenkins-hbase20.apache.org,43787,1689121924697 in 178 msec 2023-07-12 00:32:06,318 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-12 00:32:06,318 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=3d92e4b6acd18630a20a2cb252f2c7f6, ASSIGN in 341 msec 2023-07-12 00:32:06,319 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 00:32:06,319 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689121926319"}]},"ts":"1689121926319"} 2023-07-12 00:32:06,321 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-12 00:32:06,322 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-12 00:32:06,323 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:06,323 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-12 00:32:06,323 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:06,323 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 00:32:06,325 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 403 msec 2023-07-12 00:32:06,327 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-12 00:32:06,339 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 00:32:06,342 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 14 msec 2023-07-12 00:32:06,349 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-12 00:32:06,358 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 00:32:06,362 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 12 msec 2023-07-12 00:32:06,373 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 00:32:06,375 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 00:32:06,375 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 1.639sec 2023-07-12 00:32:06,375 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 00:32:06,375 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 00:32:06,375 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 00:32:06,375 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,45201,1689121924613-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 00:32:06,375 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,45201,1689121924613-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 00:32:06,378 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 00:32:06,430 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ReadOnlyZKClient(139): Connect 0x78c85624 to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:32:06,442 DEBUG [Listener at localhost.localdomain/34481] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@644229fb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:32:06,445 DEBUG [hconnection-0x737896d3-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 00:32:06,449 INFO [RS-EventLoopGroup-14-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34476, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 00:32:06,452 INFO [Listener at localhost.localdomain/34481] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,45201,1689121924613 2023-07-12 00:32:06,452 INFO [Listener at localhost.localdomain/34481] hbase.TestZooKeeper(139): Starting testMasterZKSessionRecoveryFailure 2023-07-12 00:32:06,453 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 00:32:06,453 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(159): ***** ABORTING master jenkins-hbase20.apache.org,45201,1689121924613: Test recovery from zk session expired ***** org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.TestZooKeeper.testMasterZKSessionRecoveryFailure(TestZooKeeper.java:142) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:32:06,453 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,45201,1689121924613' ***** 2023-07-12 00:32:06,453 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2311): STOPPED: Stopped by Listener at localhost.localdomain/34481 2023-07-12 00:32:06,455 DEBUG [M:0;jenkins-hbase20:45201] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8ebdc4e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-12 00:32:06,455 INFO [M:0;jenkins-hbase20:45201] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,45201,1689121924613 2023-07-12 00:32:06,455 DEBUG [M:0;jenkins-hbase20:45201] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1e8961d1 to 127.0.0.1:58965 2023-07-12 00:32:06,456 DEBUG [M:0;jenkins-hbase20:45201] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:06,456 INFO [M:0;jenkins-hbase20:45201] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,45201,1689121924613; all regions closed. 2023-07-12 00:32:06,456 DEBUG [M:0;jenkins-hbase20:45201] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:06,456 DEBUG [M:0;jenkins-hbase20:45201] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 00:32:06,456 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 00:32:06,456 DEBUG [M:0;jenkins-hbase20:45201] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 00:32:06,456 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689121924982] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689121924982,5,FailOnTimeoutGroup] 2023-07-12 00:32:06,456 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689121924979] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689121924979,5,FailOnTimeoutGroup] 2023-07-12 00:32:06,456 INFO [M:0;jenkins-hbase20:45201] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 00:32:06,457 INFO [M:0;jenkins-hbase20:45201] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 00:32:06,457 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:06,457 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:43787-0x1007e9741810017, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:06,457 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:34207-0x1007e9741810016, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:06,457 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:06,458 INFO [Listener at localhost.localdomain/34481] hbase.TestZooKeeper(154): Creating table testMasterZKSessionRecoveryFailure_1689121926458 2023-07-12 00:32:06,458 INFO [M:0;jenkins-hbase20:45201] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-07-12 00:32:06,458 DEBUG [M:0;jenkins-hbase20:45201] master.HMaster(1512): Stopping service threads 2023-07-12 00:32:06,458 INFO [M:0;jenkins-hbase20:45201] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 00:32:06,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:06,459 INFO [M:0;jenkins-hbase20:45201] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 00:32:06,458 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:06,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34207-0x1007e9741810016, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:06,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43787-0x1007e9741810017, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:06,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:06,458 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:06,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:06,461 DEBUG [zk-event-processor-pool-0] master.ActiveMasterManager(186): No master available. Notifying waiting threads 2023-07-12 00:32:06,460 ERROR [M:0;jenkins-hbase20:45201] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-5,5,PEWorkerGroup] 2023-07-12 00:32:06,462 INFO [M:0;jenkins-hbase20:45201] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=true 2023-07-12 00:32:06,462 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 00:32:06,462 DEBUG [M:0;jenkins-hbase20:45201] zookeeper.ZKUtil(398): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 00:32:06,462 WARN [M:0;jenkins-hbase20:45201] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 00:32:06,462 INFO [M:0;jenkins-hbase20:45201] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 00:32:06,463 INFO [M:0;jenkins-hbase20:45201] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=true 2023-07-12 00:32:06,469 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:34207-0x1007e9741810016, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:06,469 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:06,469 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:06,469 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:06,469 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:06,469 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:43787-0x1007e9741810017, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:06,471 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:32:06,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:32:06,471 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,46459,1689121924634 from backup master directory 2023-07-12 00:32:06,472 DEBUG [M:0;jenkins-hbase20:45201] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 00:32:06,472 INFO [M:0;jenkins-hbase20:45201] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:06,472 DEBUG [M:0;jenkins-hbase20:45201] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:06,472 DEBUG [M:0;jenkins-hbase20:45201] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 00:32:06,472 DEBUG [M:0;jenkins-hbase20:45201] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:06,472 ERROR [M:0;jenkins-hbase20:45201] regionserver.HRegion(1825): Memstore data size is 24669 in region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:06,472 INFO [M:0;jenkins-hbase20:45201] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:06,473 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:32:06,473 DEBUG [M:0;jenkins-hbase20:45201] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 00:32:06,473 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 00:32:06,473 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 00:32:06,472 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,46459,1689121924634 2023-07-12 00:32:06,473 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,46459,1689121924634 2023-07-12 00:32:06,473 INFO [M:0;jenkins-hbase20:45201] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 00:32:06,473 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:32:06,474 INFO [M:0;jenkins-hbase20:45201] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:45201 2023-07-12 00:32:06,476 DEBUG [M:0;jenkins-hbase20:45201] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,45201,1689121924613 already deleted, retry=false 2023-07-12 00:32:06,489 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:06,507 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x742c4b3c to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:32:06,509 WARN [RS-EventLoopGroup-14-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase20.apache.org/148.251.75.209:45201 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:45201 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:32:06,511 DEBUG [RS-EventLoopGroup-14-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase20.apache.org/148.251.75.209:45201 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:45201 2023-07-12 00:32:06,512 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@53fc7797, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:32:06,512 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 00:32:06,513 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 00:32:06,513 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:32:06,519 WARN [RS-EventLoopGroup-14-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase20.apache.org/148.251.75.209:45201 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:45201 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:32:06,519 DEBUG [RS-EventLoopGroup-14-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase20.apache.org/148.251.75.209:45201 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:45201 2023-07-12 00:32:06,522 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,45201,1689121924613 to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,45201,1689121924613-dead as it is dead 2023-07-12 00:32:06,522 INFO [master/jenkins-hbase20:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,45201,1689121924613-dead/jenkins-hbase20.apache.org%2C45201%2C1689121924613.1689121924832 2023-07-12 00:32:06,523 INFO [master/jenkins-hbase20:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,45201,1689121924613-dead/jenkins-hbase20.apache.org%2C45201%2C1689121924613.1689121924832 after 0ms 2023-07-12 00:32:06,524 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,45201,1689121924613-dead/jenkins-hbase20.apache.org%2C45201%2C1689121924613.1689121924832 to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase20.apache.org%2C45201%2C1689121924613.1689121924832 2023-07-12 00:32:06,524 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,45201,1689121924613-dead 2023-07-12 00:32:06,524 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,46459,1689121924634 2023-07-12 00:32:06,526 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C46459%2C1689121924634, suffix=, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,46459,1689121924634, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/oldWALs, maxLogs=10 2023-07-12 00:32:06,542 DEBUG [RS-EventLoopGroup-14-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:32:06,542 DEBUG [RS-EventLoopGroup-14-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:32:06,544 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,46459,1689121924634/jenkins-hbase20.apache.org%2C46459%2C1689121924634.1689121926527 2023-07-12 00:32:06,544 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK], DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK]] 2023-07-12 00:32:06,544 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 00:32:06,545 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:06,545 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:06,545 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:06,546 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:06,547 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 00:32:06,548 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 00:32:06,548 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:06,549 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5179): Found 1 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-07-12 00:32:06,549 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5276): Replaying edits from hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase20.apache.org%2C45201%2C1689121924613.1689121924832 2023-07-12 00:32:06,556 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5464): Applied 71, skipped 0, firstSequenceIdInLog=3, maxSequenceIdInLog=65, path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase20.apache.org%2C45201%2C1689121924613.1689121924832 2023-07-12 00:32:06,557 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=24.09 KB heapSize=29.59 KB 2023-07-12 00:32:06,564 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-12 00:32:06,567 INFO [RS-EventLoopGroup-12-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48230, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-12 00:32:06,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46459] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:1166) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:06,572 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.09 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/ac95d69848074542881378fac56dacc7 2023-07-12 00:32:06,579 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/ac95d69848074542881378fac56dacc7 as hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/ac95d69848074542881378fac56dacc7 2023-07-12 00:32:06,588 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/ac95d69848074542881378fac56dacc7, entries=71, sequenceid=65, filesize=28.9 K 2023-07-12 00:32:06,588 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(2948): Finished flush of dataSize ~24.09 KB/24669, heapSize ~29.57 KB/30280, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 32ms, sequenceid=65, compaction requested=false; wal=null 2023-07-12 00:32:06,589 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase20.apache.org%2C45201%2C1689121924613.1689121924832 2023-07-12 00:32:06,592 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:06,595 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/65.seqid, newMaxSeqId=65, maxSeqId=1 2023-07-12 00:32:06,596 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=66; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72705572, jitterRate=0.08339744806289673}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:06,596 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 00:32:06,596 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 00:32:06,597 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-12 00:32:06,597 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-12 00:32:06,598 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 00:32:06,598 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 00:32:06,602 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 00:32:06,602 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 00:32:06,602 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 00:32:06,603 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 00:32:06,603 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-07-12 00:32:06,603 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 00:32:06,604 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 00:32:06,604 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase20.apache.org,43787,1689121924697, table=hbase:meta, region=1588230740 2023-07-12 00:32:06,606 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 2 possibly 'live' servers, and 0 'splitting'. 2023-07-12 00:32:06,606 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,34207,1689121924665 2023-07-12 00:32:06,607 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,43787,1689121924697 2023-07-12 00:32:06,611 DEBUG [RS:0;jenkins-hbase20:34207] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 00:32:06,613 INFO [RS-EventLoopGroup-12-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:32793, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.4 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 00:32:06,614 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=46459] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:06,617 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 00:32:06,617 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 00:32:06,617 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 00:32:06,618 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 00:32:06,618 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 00:32:06,619 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 00:32:06,619 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 00:32:06,619 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,46459,1689121924634, sessionid=0x1007e9741810015, setting cluster-up flag (Was=true) 2023-07-12 00:32:06,620 DEBUG [RS:1;jenkins-hbase20:43787] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 00:32:06,621 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 00:32:06,621 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,46459,1689121924634 2023-07-12 00:32:06,622 INFO [RS-EventLoopGroup-12-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:35191, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.5 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 00:32:06,622 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=46459] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:06,623 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 00:32:06,624 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,46459,1689121924634 2023-07-12 00:32:06,625 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.hbase-snapshot/.tmp 2023-07-12 00:32:06,625 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:32:06,626 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:32:06,626 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:32:06,626 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:32:06,626 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-07-12 00:32:06,626 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:06,626 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-12 00:32:06,626 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:06,634 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689121956634 2023-07-12 00:32:06,634 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 00:32:06,634 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 00:32:06,635 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 00:32:06,635 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 00:32:06,635 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 00:32:06,635 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 00:32:06,638 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:06,640 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 00:32:06,640 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 00:32:06,640 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 00:32:06,640 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 00:32:06,641 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 00:32:06,642 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689121926641,5,FailOnTimeoutGroup] 2023-07-12 00:32:06,643 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689121926642,5,FailOnTimeoutGroup] 2023-07-12 00:32:06,643 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:06,643 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 00:32:06,643 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:06,643 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:06,643 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689121926643, completionTime=-1 2023-07-12 00:32:06,643 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-12 00:32:06,643 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 00:32:06,647 DEBUG [hconnection-0x113397be-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 00:32:06,648 INFO [RS-EventLoopGroup-14-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34482, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 00:32:06,650 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=3d92e4b6acd18630a20a2cb252f2c7f6, regionState=OPEN, lastHost=jenkins-hbase20.apache.org,43787,1689121924697, regionLocation=jenkins-hbase20.apache.org,43787,1689121924697, openSeqNum=2 2023-07-12 00:32:06,651 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-12 00:32:06,651 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689121986651 2023-07-12 00:32:06,651 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689122046651 2023-07-12 00:32:06,651 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 7 msec 2023-07-12 00:32:06,659 INFO [M:0;jenkins-hbase20:45201] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,45201,1689121924613; zookeeper connection closed. 2023-07-12 00:32:06,659 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:06,661 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45201-0x1007e9741810014, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:06,665 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,46459,1689121924634-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:06,665 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,46459,1689121924634-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:06,665 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,46459,1689121924634-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:06,666 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:46459, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:06,666 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:06,666 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-07-12 00:32:06,669 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 00:32:06,681 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 00:32:06,686 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 00:32:06,686 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 0.213sec 2023-07-12 00:32:06,686 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 00:32:06,686 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 00:32:06,686 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 00:32:06,686 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,46459,1689121924634-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 00:32:06,687 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,46459,1689121924634-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 00:32:06,689 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 00:32:06,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46459] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'testMasterZKSessionRecoveryFailure_1689121926458', {NAME => 'fam', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 00:32:06,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46459] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testMasterZKSessionRecoveryFailure_1689121926458 2023-07-12 00:32:06,777 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testMasterZKSessionRecoveryFailure_1689121926458 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 00:32:06,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46459] master.MasterRpcServices(700): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "testMasterZKSessionRecoveryFailure_1689121926458" procId is: 9 2023-07-12 00:32:06,778 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testMasterZKSessionRecoveryFailure_1689121926458 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 00:32:06,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46459] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 00:32:06,781 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testMasterZKSessionRecoveryFailure_1689121926458/01fbf40af1a1c4cdce64c4da8d4bc8c3 2023-07-12 00:32:06,781 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testMasterZKSessionRecoveryFailure_1689121926458/01fbf40af1a1c4cdce64c4da8d4bc8c3 empty. 2023-07-12 00:32:06,782 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testMasterZKSessionRecoveryFailure_1689121926458/01fbf40af1a1c4cdce64c4da8d4bc8c3 2023-07-12 00:32:06,782 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testMasterZKSessionRecoveryFailure_1689121926458 regions 2023-07-12 00:32:06,795 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testMasterZKSessionRecoveryFailure_1689121926458/.tabledesc/.tableinfo.0000000001 2023-07-12 00:32:06,796 INFO [RegionOpenAndInit-testMasterZKSessionRecoveryFailure_1689121926458-pool-0] regionserver.HRegion(7675): creating {ENCODED => 01fbf40af1a1c4cdce64c4da8d4bc8c3, NAME => 'testMasterZKSessionRecoveryFailure_1689121926458,,1689121926772.01fbf40af1a1c4cdce64c4da8d4bc8c3.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testMasterZKSessionRecoveryFailure_1689121926458', {NAME => 'fam', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp 2023-07-12 00:32:06,811 DEBUG [RegionOpenAndInit-testMasterZKSessionRecoveryFailure_1689121926458-pool-0] regionserver.HRegion(866): Instantiated testMasterZKSessionRecoveryFailure_1689121926458,,1689121926772.01fbf40af1a1c4cdce64c4da8d4bc8c3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:06,811 DEBUG [RegionOpenAndInit-testMasterZKSessionRecoveryFailure_1689121926458-pool-0] regionserver.HRegion(1604): Closing 01fbf40af1a1c4cdce64c4da8d4bc8c3, disabling compactions & flushes 2023-07-12 00:32:06,811 INFO [RegionOpenAndInit-testMasterZKSessionRecoveryFailure_1689121926458-pool-0] regionserver.HRegion(1626): Closing region testMasterZKSessionRecoveryFailure_1689121926458,,1689121926772.01fbf40af1a1c4cdce64c4da8d4bc8c3. 2023-07-12 00:32:06,811 DEBUG [RegionOpenAndInit-testMasterZKSessionRecoveryFailure_1689121926458-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testMasterZKSessionRecoveryFailure_1689121926458,,1689121926772.01fbf40af1a1c4cdce64c4da8d4bc8c3. 2023-07-12 00:32:06,811 DEBUG [RegionOpenAndInit-testMasterZKSessionRecoveryFailure_1689121926458-pool-0] regionserver.HRegion(1714): Acquired close lock on testMasterZKSessionRecoveryFailure_1689121926458,,1689121926772.01fbf40af1a1c4cdce64c4da8d4bc8c3. after waiting 0 ms 2023-07-12 00:32:06,812 DEBUG [RegionOpenAndInit-testMasterZKSessionRecoveryFailure_1689121926458-pool-0] regionserver.HRegion(1724): Updates disabled for region testMasterZKSessionRecoveryFailure_1689121926458,,1689121926772.01fbf40af1a1c4cdce64c4da8d4bc8c3. 2023-07-12 00:32:06,812 INFO [RegionOpenAndInit-testMasterZKSessionRecoveryFailure_1689121926458-pool-0] regionserver.HRegion(1838): Closed testMasterZKSessionRecoveryFailure_1689121926458,,1689121926772.01fbf40af1a1c4cdce64c4da8d4bc8c3. 2023-07-12 00:32:06,812 DEBUG [RegionOpenAndInit-testMasterZKSessionRecoveryFailure_1689121926458-pool-0] regionserver.HRegion(1558): Region close journal for 01fbf40af1a1c4cdce64c4da8d4bc8c3: 2023-07-12 00:32:06,814 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testMasterZKSessionRecoveryFailure_1689121926458 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 00:32:06,815 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testMasterZKSessionRecoveryFailure_1689121926458,,1689121926772.01fbf40af1a1c4cdce64c4da8d4bc8c3.","families":{"info":[{"qualifier":"regioninfo","vlen":82,"tag":[],"timestamp":"1689121926815"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689121926815"}]},"ts":"1689121926815"} 2023-07-12 00:32:06,817 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 00:32:06,818 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testMasterZKSessionRecoveryFailure_1689121926458 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 00:32:06,818 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testMasterZKSessionRecoveryFailure_1689121926458","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689121926818"}]},"ts":"1689121926818"} 2023-07-12 00:32:06,820 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testMasterZKSessionRecoveryFailure_1689121926458, state=ENABLING in hbase:meta 2023-07-12 00:32:06,822 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-07-12 00:32:06,823 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 00:32:06,823 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 00:32:06,823 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 00:32:06,823 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testMasterZKSessionRecoveryFailure_1689121926458, region=01fbf40af1a1c4cdce64c4da8d4bc8c3, ASSIGN}] 2023-07-12 00:32:06,826 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testMasterZKSessionRecoveryFailure_1689121926458, region=01fbf40af1a1c4cdce64c4da8d4bc8c3, ASSIGN 2023-07-12 00:32:06,827 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testMasterZKSessionRecoveryFailure_1689121926458, region=01fbf40af1a1c4cdce64c4da8d4bc8c3, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34207,1689121924665; forceNewPlan=false, retain=false 2023-07-12 00:32:06,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46459] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 00:32:06,978 INFO [jenkins-hbase20:46459] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-12 00:32:06,979 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=01fbf40af1a1c4cdce64c4da8d4bc8c3, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34207,1689121924665 2023-07-12 00:32:06,979 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testMasterZKSessionRecoveryFailure_1689121926458,,1689121926772.01fbf40af1a1c4cdce64c4da8d4bc8c3.","families":{"info":[{"qualifier":"regioninfo","vlen":82,"tag":[],"timestamp":"1689121926979"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121926979"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121926979"}]},"ts":"1689121926979"} 2023-07-12 00:32:06,982 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 01fbf40af1a1c4cdce64c4da8d4bc8c3, server=jenkins-hbase20.apache.org,34207,1689121924665}] 2023-07-12 00:32:07,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46459] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 00:32:07,137 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,34207,1689121924665 2023-07-12 00:32:07,138 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 00:32:07,139 INFO [RS-EventLoopGroup-13-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52680, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 00:32:07,142 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open testMasterZKSessionRecoveryFailure_1689121926458,,1689121926772.01fbf40af1a1c4cdce64c4da8d4bc8c3. 2023-07-12 00:32:07,143 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 01fbf40af1a1c4cdce64c4da8d4bc8c3, NAME => 'testMasterZKSessionRecoveryFailure_1689121926458,,1689121926772.01fbf40af1a1c4cdce64c4da8d4bc8c3.', STARTKEY => '', ENDKEY => ''} 2023-07-12 00:32:07,143 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testMasterZKSessionRecoveryFailure_1689121926458 01fbf40af1a1c4cdce64c4da8d4bc8c3 2023-07-12 00:32:07,143 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated testMasterZKSessionRecoveryFailure_1689121926458,,1689121926772.01fbf40af1a1c4cdce64c4da8d4bc8c3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:07,143 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 01fbf40af1a1c4cdce64c4da8d4bc8c3 2023-07-12 00:32:07,143 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 01fbf40af1a1c4cdce64c4da8d4bc8c3 2023-07-12 00:32:07,144 INFO [StoreOpener-01fbf40af1a1c4cdce64c4da8d4bc8c3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family fam of region 01fbf40af1a1c4cdce64c4da8d4bc8c3 2023-07-12 00:32:07,146 DEBUG [StoreOpener-01fbf40af1a1c4cdce64c4da8d4bc8c3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testMasterZKSessionRecoveryFailure_1689121926458/01fbf40af1a1c4cdce64c4da8d4bc8c3/fam 2023-07-12 00:32:07,146 DEBUG [StoreOpener-01fbf40af1a1c4cdce64c4da8d4bc8c3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testMasterZKSessionRecoveryFailure_1689121926458/01fbf40af1a1c4cdce64c4da8d4bc8c3/fam 2023-07-12 00:32:07,146 INFO [StoreOpener-01fbf40af1a1c4cdce64c4da8d4bc8c3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 01fbf40af1a1c4cdce64c4da8d4bc8c3 columnFamilyName fam 2023-07-12 00:32:07,147 INFO [StoreOpener-01fbf40af1a1c4cdce64c4da8d4bc8c3-1] regionserver.HStore(310): Store=01fbf40af1a1c4cdce64c4da8d4bc8c3/fam, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:07,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testMasterZKSessionRecoveryFailure_1689121926458/01fbf40af1a1c4cdce64c4da8d4bc8c3 2023-07-12 00:32:07,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testMasterZKSessionRecoveryFailure_1689121926458/01fbf40af1a1c4cdce64c4da8d4bc8c3 2023-07-12 00:32:07,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 01fbf40af1a1c4cdce64c4da8d4bc8c3 2023-07-12 00:32:07,152 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testMasterZKSessionRecoveryFailure_1689121926458/01fbf40af1a1c4cdce64c4da8d4bc8c3/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:07,153 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 01fbf40af1a1c4cdce64c4da8d4bc8c3; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75165936, jitterRate=0.12005972862243652}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:07,153 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 01fbf40af1a1c4cdce64c4da8d4bc8c3: 2023-07-12 00:32:07,154 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testMasterZKSessionRecoveryFailure_1689121926458,,1689121926772.01fbf40af1a1c4cdce64c4da8d4bc8c3., pid=11, masterSystemTime=1689121927137 2023-07-12 00:32:07,158 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testMasterZKSessionRecoveryFailure_1689121926458,,1689121926772.01fbf40af1a1c4cdce64c4da8d4bc8c3. 2023-07-12 00:32:07,159 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened testMasterZKSessionRecoveryFailure_1689121926458,,1689121926772.01fbf40af1a1c4cdce64c4da8d4bc8c3. 2023-07-12 00:32:07,160 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=01fbf40af1a1c4cdce64c4da8d4bc8c3, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34207,1689121924665 2023-07-12 00:32:07,160 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testMasterZKSessionRecoveryFailure_1689121926458,,1689121926772.01fbf40af1a1c4cdce64c4da8d4bc8c3.","families":{"info":[{"qualifier":"regioninfo","vlen":82,"tag":[],"timestamp":"1689121927160"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121927160"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121927160"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121927160"}]},"ts":"1689121927160"} 2023-07-12 00:32:07,163 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-12 00:32:07,163 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 01fbf40af1a1c4cdce64c4da8d4bc8c3, server=jenkins-hbase20.apache.org,34207,1689121924665 in 179 msec 2023-07-12 00:32:07,165 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-12 00:32:07,165 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testMasterZKSessionRecoveryFailure_1689121926458, region=01fbf40af1a1c4cdce64c4da8d4bc8c3, ASSIGN in 340 msec 2023-07-12 00:32:07,166 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testMasterZKSessionRecoveryFailure_1689121926458 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 00:32:07,166 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testMasterZKSessionRecoveryFailure_1689121926458","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689121927166"}]},"ts":"1689121927166"} 2023-07-12 00:32:07,167 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testMasterZKSessionRecoveryFailure_1689121926458, state=ENABLED in hbase:meta 2023-07-12 00:32:07,169 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testMasterZKSessionRecoveryFailure_1689121926458 execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 00:32:07,171 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testMasterZKSessionRecoveryFailure_1689121926458 in 397 msec 2023-07-12 00:32:07,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46459] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 00:32:07,384 INFO [Listener at localhost.localdomain/34481] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testMasterZKSessionRecoveryFailure_1689121926458, procId: 9 completed 2023-07-12 00:32:07,384 INFO [Listener at localhost.localdomain/34481] hbase.TestZooKeeper(165): Putting table testMasterZKSessionRecoveryFailure_1689121926458 2023-07-12 00:32:07,386 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 00:32:07,388 INFO [RS-EventLoopGroup-13-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52696, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 00:32:07,390 DEBUG [Listener at localhost.localdomain/34481] master.HMaster(2711): Abort called but aborted=true, stopped=true 2023-07-12 00:32:07,390 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 00:32:07,390 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): ***** ABORTING master jenkins-hbase20.apache.org,46459,1689121924634: killAll ***** 2023-07-12 00:32:07,391 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,46459,1689121924634' ***** 2023-07-12 00:32:07,391 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2311): STOPPED: Stopped by Listener at localhost.localdomain/34481 2023-07-12 00:32:07,391 DEBUG [M:1;jenkins-hbase20:46459] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38a0f2c3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-12 00:32:07,391 INFO [M:1;jenkins-hbase20:46459] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,46459,1689121924634 2023-07-12 00:32:07,391 DEBUG [M:1;jenkins-hbase20:46459] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x742c4b3c to 127.0.0.1:58965 2023-07-12 00:32:07,391 DEBUG [M:1;jenkins-hbase20:46459] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:07,391 INFO [M:1;jenkins-hbase20:46459] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,46459,1689121924634; all regions closed. 2023-07-12 00:32:07,391 DEBUG [M:1;jenkins-hbase20:46459] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:07,392 DEBUG [M:1;jenkins-hbase20:46459] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 00:32:07,392 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 00:32:07,392 DEBUG [M:1;jenkins-hbase20:46459] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 00:32:07,392 INFO [M:1;jenkins-hbase20:46459] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 00:32:07,392 INFO [M:1;jenkins-hbase20:46459] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 00:32:07,392 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689121926641] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689121926641,5,FailOnTimeoutGroup] 2023-07-12 00:32:07,392 INFO [M:1;jenkins-hbase20:46459] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-07-12 00:32:07,393 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689121926642] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689121926642,5,FailOnTimeoutGroup] 2023-07-12 00:32:07,393 DEBUG [M:1;jenkins-hbase20:46459] master.HMaster(1512): Stopping service threads 2023-07-12 00:32:07,394 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:34207-0x1007e9741810016, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:07,394 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:43787-0x1007e9741810017, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:07,394 INFO [M:1;jenkins-hbase20:46459] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 00:32:07,393 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:07,394 INFO [M:1;jenkins-hbase20:46459] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 00:32:07,394 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase20.apache.org,34207,1689121924665: killAll ***** 2023-07-12 00:32:07,394 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:07,395 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 00:32:07,395 ERROR [M:1;jenkins-hbase20:46459] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-6,5,PEWorkerGroup] 2023-07-12 00:32:07,395 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-12 00:32:07,396 INFO [M:1;jenkins-hbase20:46459] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=true 2023-07-12 00:32:07,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:07,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34207-0x1007e9741810016, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:07,396 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-12 00:32:07,396 DEBUG [M:1;jenkins-hbase20:46459] zookeeper.ZKUtil(398): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 00:32:07,396 WARN [M:1;jenkins-hbase20:46459] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 00:32:07,396 INFO [M:1;jenkins-hbase20:46459] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 00:32:07,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43787-0x1007e9741810017, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:07,397 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 00:32:07,397 INFO [M:1;jenkins-hbase20:46459] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=true 2023-07-12 00:32:07,400 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-12 00:32:07,400 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-12 00:32:07,401 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,46459,1689121924634/jenkins-hbase20.apache.org%2C46459%2C1689121924634.1689121926527 not finished, retry = 0 2023-07-12 00:32:07,402 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1047003136, "init": 524288000, "max": 2051014656, "used": 512762616 }, "NonHeapMemoryUsage": { "committed": 134307840, "init": 2555904, "max": -1, "used": 131556696 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 0, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 13538, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 0, "ProcessCallTime_num_ops": 55, "ProcessCallTime_min": 0, "ProcessCallTime_max": 8, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 2, "ProcessCallTime_75th_percentile": 2, "ProcessCallTime_90th_percentile": 3, "ProcessCallTime_95th_percentile": 7, "ProcessCallTime_98th_percentile": 8, "ProcessCallTime_99th_percentile": 8, "ProcessCallTime_99.9th_percentile": 8, "ProcessCallTime_TimeRangeCount_0-1": 11, "ProcessCallTime_TimeRangeCount_1-3": 9, "ProcessCallTime_TimeRangeCount_3-10": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 55, "QueueCallTime_min": 0, "QueueCallTime_max": 1, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 1, "QueueCallTime_99th_percentile": 1, "QueueCallTime_99.9th_percentile": 1, "QueueCallTime_TimeRangeCount_0-1": 22, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 55, "TotalCallTime_min": 0, "TotalCallTime_max": 8, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 2, "TotalCallTime_75th_percentile": 2, "TotalCallTime_90th_percentile": 3, "TotalCallTime_95th_percentile": 7, "TotalCallTime_98th_percentile": 8, "TotalCallTime_99th_percentile": 8, "TotalCallTime_99.9th_percentile": 8, "TotalCallTime_TimeRangeCount_0-1": 10, "TotalCallTime_TimeRangeCount_1-3": 10, "TotalCallTime_TimeRangeCount_3-10": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 55, "ResponseSize_min": 0, "ResponseSize_max": 983, "ResponseSize_mean": 127, "ResponseSize_25th_percentile": 2, "ResponseSize_median": 4, "ResponseSize_75th_percentile": 22, "ResponseSize_90th_percentile": 449, "ResponseSize_95th_percentile": 716, "ResponseSize_98th_percentile": 876, "ResponseSize_99th_percentile": 929, "ResponseSize_99.9th_percentile": 977, "ResponseSize_SizeRangeCount_0-10": 15, "ResponseSize_SizeRangeCount_10-100": 3, "ResponseSize_SizeRangeCount_100-1000": 4, "exceptions.UnknownScannerException": 0, "exceptions": 0, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 0, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 55, "RequestSize_min": 80, "RequestSize_max": 340, "RequestSize_mean": 143, "RequestSize_25th_percentile": 97, "RequestSize_median": 116, "RequestSize_75th_percentile": 150, "RequestSize_90th_percentile": 216, "RequestSize_95th_percentile": 280, "RequestSize_98th_percentile": 339, "RequestSize_99th_percentile": 339, "RequestSize_99.9th_percentile": 339, "RequestSize_SizeRangeCount_10-100": 8, "RequestSize_SizeRangeCount_100-1000": 14, "sentBytes": 5103 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:58965", "tag.serverName": "jenkins-hbase20.apache.org,42559,1689121902528", "tag.clusterId": "b94c042f-d510-47b7-9d4c-011dde64d627", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "regionCount": 0, "storeCount": 0, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 0, "maxStoreFileCount": 0, "memStoreSize": 0, "storeFileSize": 0, "maxStoreFileAge": 0, "minStoreFileAge": 3067, "avgStoreFileAge": 3537, "numReferenceFiles": 0, "regionServerStartTime": 1689121902528, "averageRegionSize": 6136, "storeFileIndexSize": 0, "staticIndexSize": 0, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 3948, "localBytesRead": 3948, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803968, "blockCacheCount": 0, "blockCacheDataBlockCount": 0, "blockCacheSize": 601920, "blockCacheCountHitPercent": 0.0, "blockCacheExpressHitPercent": 0.0, "l1CacheSize": 601920, "l1CacheFreeSize": 819803968, "l1CacheCount": 0, "l1CacheEvictionCount": 0, "l1CacheHitCount": 0, "l1CacheMissCount": 0, "l1CacheHitRatio": 0.0, "l1CacheMissRatio": 0.0, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 4608, "ByteBuffAllocatorPoolAllocationBytes": 133120, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 10, "totalRowActionRequestCount": 0, "readRequestCount": 0, "filteredReadRequestCount": 0, "writeRequestCount": 0, "rpcGetRequestCount": 1, "rpcFullScanRequestCount": 0, "rpcScanRequestCount": 1, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 8, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 0, "blockCacheHitCountPrimary": 0, "blockCacheHitCachingCount": 0, "blockCacheMissCount": 0, "blockCacheMissCountPrimary": 0, "blockCacheMissCachingCount": 0, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 0, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 0, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 0, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 0, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 1, "ScanTime_min": 1, "ScanTime_max": 1, "ScanTime_mean": 1, "ScanTime_25th_percentile": 1, "ScanTime_median": 1, "ScanTime_75th_percentile": 1, "ScanTime_90th_percentile": 1, "ScanTime_95th_percentile": 1, "ScanTime_98th_percentile": 1, "ScanTime_99th_percentile": 1, "ScanTime_99.9th_percentile": 1, "ScanTime_TimeRangeCount_0-1": 1, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 8, "Put_min": 2, "Put_max": 2, "Put_mean": 2, "Put_25th_percentile": 2, "Put_median": 2, "Put_75th_percentile": 2, "Put_90th_percentile": 2, "Put_95th_percentile": 2, "Put_98th_percentile": 2, "Put_99th_percentile": 2, "Put_99.9th_percentile": 2, "Put_TimeRangeCount_1-3": 1, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 2, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 1, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 15699, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 2, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 1, "ScanSize_min": 956, "ScanSize_max": 956, "ScanSize_mean": 956, "ScanSize_25th_percentile": 956, "ScanSize_median": 956, "ScanSize_75th_percentile": 956, "ScanSize_90th_percentile": 956, "ScanSize_95th_percentile": 956, "ScanSize_98th_percentile": 956, "ScanSize_99th_percentile": 956, "ScanSize_99.9th_percentile": 956, "ScanSize_SizeRangeCount_100-1000": 1, "slowGetCount": 0, "flushedMemstoreBytes": 1370, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 2, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 28, "ServerReadQueryPerSecond_mean_rate": 1.5853034410975209, "ServerReadQueryPerSecond_1min_rate": 2.4394406617012336, "ServerReadQueryPerSecond_5min_rate": 2.5642928612206086, "ServerReadQueryPerSecond_15min_rate": 2.587885329492682, "ServerWriteQueryPerSecond_count": 22, "ServerWriteQueryPerSecond_mean_rate": 1.2455953139609977, "ServerWriteQueryPerSecond_1min_rate": 1.4159911170741353, "ServerWriteQueryPerSecond_5min_rate": 1.4033057092356764, "ServerWriteQueryPerSecond_15min_rate": 1.4011080303990207 } ] } 2023-07-12 00:32:07,403 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46459] master.MasterRpcServices(612): jenkins-hbase20.apache.org,34207,1689121924665 reported a fatal error: ***** ABORTING region server jenkins-hbase20.apache.org,34207,1689121924665: killAll ***** 2023-07-12 00:32:07,406 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,34207,1689121924665' ***** 2023-07-12 00:32:07,406 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2311): STOPPED: killAll 2023-07-12 00:32:07,406 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase20.apache.org,43787,1689121924697: killAll ***** 2023-07-12 00:32:07,406 INFO [RS:0;jenkins-hbase20:34207] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 00:32:07,406 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 00:32:07,406 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 00:32:07,406 INFO [RS:0;jenkins-hbase20:34207] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-12 00:32:07,406 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-12 00:32:07,407 INFO [RS:0;jenkins-hbase20:34207] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-12 00:32:07,407 INFO [RS:0;jenkins-hbase20:34207] regionserver.HRegionServer(3305): Received CLOSE for 01fbf40af1a1c4cdce64c4da8d4bc8c3 2023-07-12 00:32:07,408 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-12 00:32:07,408 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-12 00:32:07,408 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-12 00:32:07,409 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1047003136, "init": 524288000, "max": 2051014656, "used": 513027160 }, "NonHeapMemoryUsage": { "committed": 134307840, "init": 2555904, "max": -1, "used": 131578224 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 0, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 13538, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 0, "ProcessCallTime_num_ops": 55, "ProcessCallTime_min": 0, "ProcessCallTime_max": 8, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 2, "ProcessCallTime_75th_percentile": 2, "ProcessCallTime_90th_percentile": 3, "ProcessCallTime_95th_percentile": 7, "ProcessCallTime_98th_percentile": 8, "ProcessCallTime_99th_percentile": 8, "ProcessCallTime_99.9th_percentile": 8, "ProcessCallTime_TimeRangeCount_0-1": 11, "ProcessCallTime_TimeRangeCount_1-3": 9, "ProcessCallTime_TimeRangeCount_3-10": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 55, "QueueCallTime_min": 0, "QueueCallTime_max": 1, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 1, "QueueCallTime_99th_percentile": 1, "QueueCallTime_99.9th_percentile": 1, "QueueCallTime_TimeRangeCount_0-1": 22, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 55, "TotalCallTime_min": 0, "TotalCallTime_max": 8, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 2, "TotalCallTime_75th_percentile": 2, "TotalCallTime_90th_percentile": 3, "TotalCallTime_95th_percentile": 7, "TotalCallTime_98th_percentile": 8, "TotalCallTime_99th_percentile": 8, "TotalCallTime_99.9th_percentile": 8, "TotalCallTime_TimeRangeCount_0-1": 10, "TotalCallTime_TimeRangeCount_1-3": 10, "TotalCallTime_TimeRangeCount_3-10": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 55, "ResponseSize_min": 0, "ResponseSize_max": 983, "ResponseSize_mean": 127, "ResponseSize_25th_percentile": 2, "ResponseSize_median": 4, "ResponseSize_75th_percentile": 22, "ResponseSize_90th_percentile": 449, "ResponseSize_95th_percentile": 716, "ResponseSize_98th_percentile": 876, "ResponseSize_99th_percentile": 929, "ResponseSize_99.9th_percentile": 977, "ResponseSize_SizeRangeCount_0-10": 15, "ResponseSize_SizeRangeCount_10-100": 3, "ResponseSize_SizeRangeCount_100-1000": 4, "exceptions.UnknownScannerException": 0, "exceptions": 0, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 0, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 55, "RequestSize_min": 80, "RequestSize_max": 340, "RequestSize_mean": 143, "RequestSize_25th_percentile": 97, "RequestSize_median": 116, "RequestSize_75th_percentile": 150, "RequestSize_90th_percentile": 216, "RequestSize_95th_percentile": 280, "RequestSize_98th_percentile": 339, "RequestSize_99th_percentile": 339, "RequestSize_99.9th_percentile": 339, "RequestSize_SizeRangeCount_10-100": 8, "RequestSize_SizeRangeCount_100-1000": 14, "sentBytes": 5103 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:58965", "tag.serverName": "jenkins-hbase20.apache.org,42559,1689121902528", "tag.clusterId": "b94c042f-d510-47b7-9d4c-011dde64d627", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "regionCount": 0, "storeCount": 0, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 0, "maxStoreFileCount": 0, "memStoreSize": 0, "storeFileSize": 0, "maxStoreFileAge": 0, "minStoreFileAge": 3067, "avgStoreFileAge": 3537, "numReferenceFiles": 0, "regionServerStartTime": 1689121902528, "averageRegionSize": 6136, "storeFileIndexSize": 0, "staticIndexSize": 0, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 3948, "localBytesRead": 3948, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803968, "blockCacheCount": 0, "blockCacheDataBlockCount": 0, "blockCacheSize": 601920, "blockCacheCountHitPercent": 0.0, "blockCacheExpressHitPercent": 0.0, "l1CacheSize": 601920, "l1CacheFreeSize": 819803968, "l1CacheCount": 0, "l1CacheEvictionCount": 0, "l1CacheHitCount": 0, "l1CacheMissCount": 0, "l1CacheHitRatio": 0.0, "l1CacheMissRatio": 0.0, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 4608, "ByteBuffAllocatorPoolAllocationBytes": 133120, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 10, "totalRowActionRequestCount": 0, "readRequestCount": 0, "filteredReadRequestCount": 0, "writeRequestCount": 0, "rpcGetRequestCount": 1, "rpcFullScanRequestCount": 0, "rpcScanRequestCount": 1, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 8, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 0, "blockCacheHitCountPrimary": 0, "blockCacheHitCachingCount": 0, "blockCacheMissCount": 0, "blockCacheMissCountPrimary": 0, "blockCacheMissCachingCount": 0, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 0, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 0, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 0, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 0, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 1, "ScanTime_min": 1, "ScanTime_max": 1, "ScanTime_mean": 1, "ScanTime_25th_percentile": 1, "ScanTime_median": 1, "ScanTime_75th_percentile": 1, "ScanTime_90th_percentile": 1, "ScanTime_95th_percentile": 1, "ScanTime_98th_percentile": 1, "ScanTime_99th_percentile": 1, "ScanTime_99.9th_percentile": 1, "ScanTime_TimeRangeCount_0-1": 1, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 8, "Put_min": 2, "Put_max": 2, "Put_mean": 2, "Put_25th_percentile": 2, "Put_median": 2, "Put_75th_percentile": 2, "Put_90th_percentile": 2, "Put_95th_percentile": 2, "Put_98th_percentile": 2, "Put_99th_percentile": 2, "Put_99.9th_percentile": 2, "Put_TimeRangeCount_1-3": 1, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 2, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 1, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 15699, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 2, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 1, "ScanSize_min": 956, "ScanSize_max": 956, "ScanSize_mean": 956, "ScanSize_25th_percentile": 956, "ScanSize_median": 956, "ScanSize_75th_percentile": 956, "ScanSize_90th_percentile": 956, "ScanSize_95th_percentile": 956, "ScanSize_98th_percentile": 956, "ScanSize_99th_percentile": 956, "ScanSize_99.9th_percentile": 956, "ScanSize_SizeRangeCount_100-1000": 1, "slowGetCount": 0, "flushedMemstoreBytes": 1370, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 2, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 28, "ServerReadQueryPerSecond_mean_rate": 1.5853034410975209, "ServerReadQueryPerSecond_1min_rate": 2.4394406617012336, "ServerReadQueryPerSecond_5min_rate": 2.5642928612206086, "ServerReadQueryPerSecond_15min_rate": 2.587885329492682, "ServerWriteQueryPerSecond_count": 22, "ServerWriteQueryPerSecond_mean_rate": 1.2455953139609977, "ServerWriteQueryPerSecond_1min_rate": 1.4159911170741353, "ServerWriteQueryPerSecond_5min_rate": 1.4033057092356764, "ServerWriteQueryPerSecond_15min_rate": 1.4011080303990207 } ] } 2023-07-12 00:32:07,410 INFO [RS:0;jenkins-hbase20:34207] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,34207,1689121924665 2023-07-12 00:32:07,410 DEBUG [RS:0;jenkins-hbase20:34207] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x615b8eac to 127.0.0.1:58965 2023-07-12 00:32:07,410 DEBUG [RS:0;jenkins-hbase20:34207] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:07,411 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 01fbf40af1a1c4cdce64c4da8d4bc8c3, disabling compactions & flushes 2023-07-12 00:32:07,411 INFO [RS:0;jenkins-hbase20:34207] regionserver.HRegionServer(1474): Waiting on 1 regions to close 2023-07-12 00:32:07,411 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region testMasterZKSessionRecoveryFailure_1689121926458,,1689121926772.01fbf40af1a1c4cdce64c4da8d4bc8c3. 2023-07-12 00:32:07,411 DEBUG [RS:0;jenkins-hbase20:34207] regionserver.HRegionServer(1478): Online Regions={01fbf40af1a1c4cdce64c4da8d4bc8c3=testMasterZKSessionRecoveryFailure_1689121926458,,1689121926772.01fbf40af1a1c4cdce64c4da8d4bc8c3.} 2023-07-12 00:32:07,411 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testMasterZKSessionRecoveryFailure_1689121926458,,1689121926772.01fbf40af1a1c4cdce64c4da8d4bc8c3. 2023-07-12 00:32:07,411 DEBUG [RS:0;jenkins-hbase20:34207] regionserver.HRegionServer(1504): Waiting on 01fbf40af1a1c4cdce64c4da8d4bc8c3 2023-07-12 00:32:07,411 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on testMasterZKSessionRecoveryFailure_1689121926458,,1689121926772.01fbf40af1a1c4cdce64c4da8d4bc8c3. after waiting 0 ms 2023-07-12 00:32:07,411 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region testMasterZKSessionRecoveryFailure_1689121926458,,1689121926772.01fbf40af1a1c4cdce64c4da8d4bc8c3. 2023-07-12 00:32:07,411 WARN [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=46459] master.MasterRpcServices(612): jenkins-hbase20.apache.org,43787,1689121924697 reported a fatal error: ***** ABORTING region server jenkins-hbase20.apache.org,43787,1689121924697: killAll ***** 2023-07-12 00:32:07,412 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1825): Memstore data size is 41 in region testMasterZKSessionRecoveryFailure_1689121926458,,1689121926772.01fbf40af1a1c4cdce64c4da8d4bc8c3. 2023-07-12 00:32:07,412 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed testMasterZKSessionRecoveryFailure_1689121926458,,1689121926772.01fbf40af1a1c4cdce64c4da8d4bc8c3. 2023-07-12 00:32:07,412 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 01fbf40af1a1c4cdce64c4da8d4bc8c3: 2023-07-12 00:32:07,412 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed testMasterZKSessionRecoveryFailure_1689121926458,,1689121926772.01fbf40af1a1c4cdce64c4da8d4bc8c3. 2023-07-12 00:32:07,413 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,43787,1689121924697' ***** 2023-07-12 00:32:07,413 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2311): STOPPED: killAll 2023-07-12 00:32:07,413 INFO [Listener at localhost.localdomain/34481] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-12 00:32:07,413 INFO [RS:1;jenkins-hbase20:43787] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 00:32:07,413 INFO [RS:1;jenkins-hbase20:43787] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-12 00:32:07,413 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 00:32:07,413 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78c85624 to 127.0.0.1:58965 2023-07-12 00:32:07,413 INFO [RS:1;jenkins-hbase20:43787] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-12 00:32:07,414 DEBUG [Listener at localhost.localdomain/34481] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:07,414 INFO [RS:1;jenkins-hbase20:43787] regionserver.HRegionServer(3305): Received CLOSE for 3d92e4b6acd18630a20a2cb252f2c7f6 2023-07-12 00:32:07,414 DEBUG [Listener at localhost.localdomain/34481] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-12 00:32:07,414 INFO [RS:1;jenkins-hbase20:43787] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,43787,1689121924697 2023-07-12 00:32:07,414 DEBUG [RS:1;jenkins-hbase20:43787] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x27f7c753 to 127.0.0.1:58965 2023-07-12 00:32:07,414 DEBUG [RS:1;jenkins-hbase20:43787] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:07,415 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 3d92e4b6acd18630a20a2cb252f2c7f6, disabling compactions & flushes 2023-07-12 00:32:07,415 INFO [RS:1;jenkins-hbase20:43787] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 00:32:07,415 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689121925920.3d92e4b6acd18630a20a2cb252f2c7f6. 2023-07-12 00:32:07,415 INFO [RS:1;jenkins-hbase20:43787] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 00:32:07,415 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689121925920.3d92e4b6acd18630a20a2cb252f2c7f6. 2023-07-12 00:32:07,415 INFO [RS:1;jenkins-hbase20:43787] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 00:32:07,415 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689121925920.3d92e4b6acd18630a20a2cb252f2c7f6. after waiting 0 ms 2023-07-12 00:32:07,415 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689121925920.3d92e4b6acd18630a20a2cb252f2c7f6. 2023-07-12 00:32:07,415 INFO [RS:1;jenkins-hbase20:43787] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-12 00:32:07,415 INFO [RS:1;jenkins-hbase20:43787] regionserver.HRegionServer(1474): Waiting on 2 regions to close 2023-07-12 00:32:07,415 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689121925920.3d92e4b6acd18630a20a2cb252f2c7f6. 2023-07-12 00:32:07,416 DEBUG [RS:1;jenkins-hbase20:43787] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740, 3d92e4b6acd18630a20a2cb252f2c7f6=hbase:namespace,,1689121925920.3d92e4b6acd18630a20a2cb252f2c7f6.} 2023-07-12 00:32:07,416 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689121925920.3d92e4b6acd18630a20a2cb252f2c7f6. 2023-07-12 00:32:07,416 DEBUG [RS:1;jenkins-hbase20:43787] regionserver.HRegionServer(1504): Waiting on 1588230740, 3d92e4b6acd18630a20a2cb252f2c7f6 2023-07-12 00:32:07,416 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 00:32:07,416 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 3d92e4b6acd18630a20a2cb252f2c7f6: 2023-07-12 00:32:07,416 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 00:32:07,416 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 00:32:07,416 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689121925920.3d92e4b6acd18630a20a2cb252f2c7f6. 2023-07-12 00:32:07,416 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 00:32:07,416 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 00:32:07,417 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1825): Memstore data size is 3103 in region hbase:meta,,1.1588230740 2023-07-12 00:32:07,417 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 00:32:07,417 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 00:32:07,417 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 00:32:07,417 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-12 00:32:07,426 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 00:32:07,440 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 00:32:07,503 DEBUG [M:1;jenkins-hbase20:46459] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 00:32:07,503 INFO [M:1;jenkins-hbase20:46459] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:07,503 DEBUG [M:1;jenkins-hbase20:46459] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:07,503 DEBUG [M:1;jenkins-hbase20:46459] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 00:32:07,503 DEBUG [M:1;jenkins-hbase20:46459] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:07,504 ERROR [M:1;jenkins-hbase20:46459] regionserver.HRegion(1825): Memstore data size is 14615 in region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:07,505 INFO [M:1;jenkins-hbase20:46459] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:07,505 DEBUG [M:1;jenkins-hbase20:46459] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 00:32:07,505 INFO [M:1;jenkins-hbase20:46459] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 00:32:07,505 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 00:32:07,505 INFO [M:1;jenkins-hbase20:46459] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:46459 2023-07-12 00:32:07,507 DEBUG [M:1;jenkins-hbase20:46459] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,46459,1689121924634 already deleted, retry=false 2023-07-12 00:32:07,611 INFO [RS:0;jenkins-hbase20:34207] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,34207,1689121924665; all regions closed. 2023-07-12 00:32:07,616 DEBUG [RS:0;jenkins-hbase20:34207] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:07,616 INFO [RS:0;jenkins-hbase20:34207] regionserver.LeaseManager(133): Closed leases 2023-07-12 00:32:07,616 INFO [RS:0;jenkins-hbase20:34207] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-12 00:32:07,616 INFO [RS:0;jenkins-hbase20:34207] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 00:32:07,616 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 00:32:07,616 INFO [RS:0;jenkins-hbase20:34207] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 00:32:07,616 INFO [RS:0;jenkins-hbase20:34207] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 00:32:07,616 INFO [RS:1;jenkins-hbase20:43787] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,43787,1689121924697; all regions closed. 2023-07-12 00:32:07,617 INFO [RS:0;jenkins-hbase20:34207] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:34207 2023-07-12 00:32:07,619 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:34207-0x1007e9741810016, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,34207,1689121924665 2023-07-12 00:32:07,619 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:34207-0x1007e9741810016, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:07,620 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:43787-0x1007e9741810017, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,34207,1689121924665 2023-07-12 00:32:07,621 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:43787-0x1007e9741810017, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:07,627 DEBUG [RS:1;jenkins-hbase20:43787] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:07,627 INFO [RS:1;jenkins-hbase20:43787] regionserver.LeaseManager(133): Closed leases 2023-07-12 00:32:07,627 INFO [RS:1;jenkins-hbase20:43787] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-12 00:32:07,627 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 00:32:07,628 INFO [RS:1;jenkins-hbase20:43787] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:43787 2023-07-12 00:32:07,629 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:43787-0x1007e9741810017, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,43787,1689121924697 2023-07-12 00:32:07,899 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:43787-0x1007e9741810017, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:07,899 INFO [RS:1;jenkins-hbase20:43787] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,43787,1689121924697; zookeeper connection closed. 2023-07-12 00:32:07,899 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:43787-0x1007e9741810017, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:07,901 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6e92289a] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6e92289a 2023-07-12 00:32:07,999 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:34207-0x1007e9741810016, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:07,999 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:34207-0x1007e9741810016, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:07,999 INFO [RS:0;jenkins-hbase20:34207] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,34207,1689121924665; zookeeper connection closed. 2023-07-12 00:32:08,000 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@3862aed5] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@3862aed5 2023-07-12 00:32:08,000 INFO [Listener at localhost.localdomain/34481] util.JVMClusterUtil(335): Shutdown of 2 master(s) and 2 regionserver(s) complete 2023-07-12 00:32:08,099 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:08,100 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:46459-0x1007e9741810015, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:08,099 INFO [M:1;jenkins-hbase20:46459] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,46459,1689121924634; zookeeper connection closed. 2023-07-12 00:32:08,202 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810013, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:08,203 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:08,205 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=testing utility connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:32:08,208 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility0x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:32:08,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): testing utility-0x1007e974181001d connected 2023-07-12 00:32:08,209 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2023-07-12 00:32:08,210 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs 2023-07-12 00:32:08,210 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/splitWAL 2023-07-12 00:32:08,210 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters 2023-07-12 00:32:08,210 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc 2023-07-12 00:32:08,211 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master-maintenance 2023-07-12 00:32:08,211 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot 2023-07-12 00:32:08,211 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/switch 2023-07-12 00:32:08,212 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-12 00:32:08,212 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/draining 2023-07-12 00:32:08,212 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-07-12 00:32:08,213 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/hbaseid 2023-07-12 00:32:08,213 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/table 2023-07-12 00:32:08,214 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/abort 2023-07-12 00:32:08,215 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired 2023-07-12 00:32:08,215 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached 2023-07-12 00:32:08,215 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/abort 2023-07-12 00:32:08,216 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired 2023-07-12 00:32:08,216 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached 2023-07-12 00:32:08,217 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/default 2023-07-12 00:32:08,217 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/hbase 2023-07-12 00:32:08,218 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/table/hbase:meta 2023-07-12 00:32:08,218 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/table/testMasterZKSessionRecoveryFailure_1689121926458 2023-07-12 00:32:08,219 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/table/hbase:namespace 2023-07-12 00:32:08,223 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table/hbase:namespace 2023-07-12 00:32:08,223 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/table 2023-07-12 00:32:08,223 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table/testMasterZKSessionRecoveryFailure_1689121926458 2023-07-12 00:32:08,223 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table/hbase:meta 2023-07-12 00:32:08,223 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 00:32:08,223 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 00:32:08,223 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 00:32:08,223 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached 2023-07-12 00:32:08,223 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot 2023-07-12 00:32:08,223 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 00:32:08,223 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 00:32:08,223 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached 2023-07-12 00:32:08,223 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc 2023-07-12 00:32:08,223 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 00:32:08,223 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 00:32:08,223 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table 2023-07-12 00:32:08,223 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:08,223 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/hbaseid 2023-07-12 00:32:08,223 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace 2023-07-12 00:32:08,223 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/draining 2023-07-12 00:32:08,226 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 00:32:08,226 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/switch 2023-07-12 00:32:08,226 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot 2023-07-12 00:32:08,226 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master-maintenance 2023-07-12 00:32:08,226 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc 2023-07-12 00:32:08,226 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:32:08,226 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL 2023-07-12 00:32:08,226 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:08,226 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 00:32:08,226 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase 2023-07-12 00:32:08,239 INFO [Listener at localhost.localdomain/34481] hbase.ResourceChecker(175): after: TestZooKeeper#testMasterZKSessionRecoveryFailure Thread=323 (was 293) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_953367935_17 at /127.0.0.1:56102 [Waiting for operation #12] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-11-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.5@localhost.localdomain:44295 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-13-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.4@localhost.localdomain:44295 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-12-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1879248547) connection to localhost.localdomain/127.0.0.1:44295 from jenkins.hfs.5 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1628992759_17 at /127.0.0.1:47422 [Waiting for operation #15] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1879248547) connection to localhost.localdomain/127.0.0.1:44295 from jenkins.hfs.4 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-13-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-11-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-13-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-12-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-11-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-12-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=682 (was 644) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=194 (was 176) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=5390 (was 5658) 2023-07-12 00:32:08,251 INFO [Listener at localhost.localdomain/34481] hbase.ResourceChecker(147): before: TestZooKeeper#testLogSplittingAfterMasterRecoveryDueToZKExpiry Thread=323, OpenFileDescriptor=682, MaxFileDescriptor=60000, SystemLoadAverage=194, ProcessCount=172, AvailableMemoryMB=5389 2023-07-12 00:32:08,262 INFO [Listener at localhost.localdomain/34481] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983 with version=8 2023-07-12 00:32:08,262 INFO [Listener at localhost.localdomain/34481] hbase.HBaseTestingUtility(1408): The hbase.fs.tmp.dir is set to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/hbase-staging 2023-07-12 00:32:08,264 INFO [Listener at localhost.localdomain/34481] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-07-12 00:32:08,264 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:08,264 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:08,264 INFO [Listener at localhost.localdomain/34481] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 00:32:08,264 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:08,265 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 00:32:08,265 INFO [Listener at localhost.localdomain/34481] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 00:32:08,265 INFO [Listener at localhost.localdomain/34481] ipc.NettyRpcServer(120): Bind to /148.251.75.209:42265 2023-07-12 00:32:08,266 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:08,267 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:08,267 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=master:42265 connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:32:08,271 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:422650x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:32:08,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:42265-0x1007e974181001e connected 2023-07-12 00:32:08,278 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:08,279 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 00:32:08,279 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 00:32:08,280 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42265 2023-07-12 00:32:08,280 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42265 2023-07-12 00:32:08,284 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42265 2023-07-12 00:32:08,284 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42265 2023-07-12 00:32:08,285 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42265 2023-07-12 00:32:08,285 INFO [Listener at localhost.localdomain/34481] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983, hbase.cluster.distributed=false 2023-07-12 00:32:08,289 INFO [Listener at localhost.localdomain/34481] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-07-12 00:32:08,289 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:08,289 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:08,289 INFO [Listener at localhost.localdomain/34481] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 00:32:08,289 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:08,289 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 00:32:08,289 INFO [Listener at localhost.localdomain/34481] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 00:32:08,290 INFO [Listener at localhost.localdomain/34481] ipc.NettyRpcServer(120): Bind to /148.251.75.209:45743 2023-07-12 00:32:08,290 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:08,291 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:08,292 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=master:45743 connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:32:08,295 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:457430x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:32:08,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:45743-0x1007e974181001f connected 2023-07-12 00:32:08,298 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:08,298 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 00:32:08,299 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 00:32:08,302 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45743 2023-07-12 00:32:08,303 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45743 2023-07-12 00:32:08,305 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45743 2023-07-12 00:32:08,306 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45743 2023-07-12 00:32:08,306 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45743 2023-07-12 00:32:08,306 INFO [Listener at localhost.localdomain/34481] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983, hbase.cluster.distributed=false 2023-07-12 00:32:08,320 INFO [Listener at localhost.localdomain/34481] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-07-12 00:32:08,320 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:08,320 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:08,321 INFO [Listener at localhost.localdomain/34481] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 00:32:08,321 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:08,321 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 00:32:08,321 INFO [Listener at localhost.localdomain/34481] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 00:32:08,322 INFO [Listener at localhost.localdomain/34481] ipc.NettyRpcServer(120): Bind to /148.251.75.209:42069 2023-07-12 00:32:08,322 INFO [Listener at localhost.localdomain/34481] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 00:32:08,323 DEBUG [Listener at localhost.localdomain/34481] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 00:32:08,324 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:08,325 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:08,326 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:42069 connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:32:08,329 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:420690x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:32:08,330 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:420690x0, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:08,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:42069-0x1007e9741810020 connected 2023-07-12 00:32:08,332 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 00:32:08,332 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 00:32:08,335 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42069 2023-07-12 00:32:08,335 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42069 2023-07-12 00:32:08,336 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42069 2023-07-12 00:32:08,336 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42069 2023-07-12 00:32:08,337 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42069 2023-07-12 00:32:08,350 INFO [Listener at localhost.localdomain/34481] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-07-12 00:32:08,350 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:08,350 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:08,350 INFO [Listener at localhost.localdomain/34481] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 00:32:08,350 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:08,350 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 00:32:08,350 INFO [Listener at localhost.localdomain/34481] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 00:32:08,352 INFO [Listener at localhost.localdomain/34481] ipc.NettyRpcServer(120): Bind to /148.251.75.209:33475 2023-07-12 00:32:08,352 INFO [Listener at localhost.localdomain/34481] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 00:32:08,354 DEBUG [Listener at localhost.localdomain/34481] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 00:32:08,355 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:08,356 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:08,357 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33475 connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:32:08,361 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:334750x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:32:08,362 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:334750x0, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:08,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:33475-0x1007e9741810021 connected 2023-07-12 00:32:08,363 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:33475-0x1007e9741810021, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 00:32:08,363 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:33475-0x1007e9741810021, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 00:32:08,364 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33475 2023-07-12 00:32:08,364 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33475 2023-07-12 00:32:08,365 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33475 2023-07-12 00:32:08,365 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33475 2023-07-12 00:32:08,365 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33475 2023-07-12 00:32:08,372 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,42265,1689121928263 2023-07-12 00:32:08,372 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,45743,1689121928288 2023-07-12 00:32:08,377 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:32:08,377 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:32:08,378 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,42265,1689121928263 2023-07-12 00:32:08,378 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,45743,1689121928288 2023-07-12 00:32:08,378 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:33475-0x1007e9741810021, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:08,379 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:08,379 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:08,379 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:08,379 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:08,379 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:08,381 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:32:08,382 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:32:08,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:32:08,383 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(278): Another master is the active master, jenkins-hbase20.apache.org,42265,1689121928263; waiting to become the next active master 2023-07-12 00:32:08,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:32:08,384 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,42265,1689121928263 from backup master directory 2023-07-12 00:32:08,385 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:32:08,385 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,42265,1689121928263 2023-07-12 00:32:08,385 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 00:32:08,385 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:32:08,385 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,42265,1689121928263 2023-07-12 00:32:08,397 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/hbase.id with ID: b855a1de-f551-45f5-8b9b-f7b8ff57b75f 2023-07-12 00:32:08,410 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:08,412 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:08,412 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:08,422 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x2efaee1a to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:32:08,432 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f8f112a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:32:08,432 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 00:32:08,433 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 00:32:08,433 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:32:08,435 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store-tmp 2023-07-12 00:32:08,449 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:08,449 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 00:32:08,449 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:08,449 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:08,449 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 00:32:08,449 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:08,449 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:08,449 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 00:32:08,450 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,42265,1689121928263 2023-07-12 00:32:08,453 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C42265%2C1689121928263, suffix=, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,42265,1689121928263, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/oldWALs, maxLogs=10 2023-07-12 00:32:08,468 DEBUG [RS-EventLoopGroup-18-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:32:08,468 DEBUG [RS-EventLoopGroup-18-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:32:08,470 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,42265,1689121928263/jenkins-hbase20.apache.org%2C42265%2C1689121928263.1689121928453 2023-07-12 00:32:08,470 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK], DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK]] 2023-07-12 00:32:08,470 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 00:32:08,470 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:08,470 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:08,470 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:08,473 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:08,475 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 00:32:08,476 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 00:32:08,477 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:08,479 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:08,480 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:08,483 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:08,485 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:08,485 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69584913, jitterRate=0.036896005272865295}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:08,486 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 00:32:08,486 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 00:32:08,487 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-12 00:32:08,487 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-12 00:32:08,487 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 00:32:08,488 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 00:32:08,488 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-07-12 00:32:08,489 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 00:32:08,491 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-12 00:32:08,492 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-12 00:32:08,503 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 00:32:08,503 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 00:32:08,504 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 00:32:08,504 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 00:32:08,508 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:08,508 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:08,509 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 00:32:08,509 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 00:32:08,510 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 00:32:08,510 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:33475-0x1007e9741810021, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 00:32:08,510 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 00:32:08,510 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 00:32:08,511 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:08,510 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 00:32:08,511 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:08,511 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,42265,1689121928263, sessionid=0x1007e974181001e, setting cluster-up flag (Was=false) 2023-07-12 00:32:08,514 INFO [M:1;jenkins-hbase20:45743] regionserver.HRegionServer(951): ClusterId : b855a1de-f551-45f5-8b9b-f7b8ff57b75f 2023-07-12 00:32:08,514 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:08,514 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:08,516 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 00:32:08,516 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,42265,1689121928263 2023-07-12 00:32:08,518 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:08,518 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:08,520 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 00:32:08,520 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,42265,1689121928263 2023-07-12 00:32:08,521 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.hbase-snapshot/.tmp 2023-07-12 00:32:08,523 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-12 00:32:08,523 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:32:08,523 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:32:08,523 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:32:08,523 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:32:08,523 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-07-12 00:32:08,523 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:08,523 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-12 00:32:08,523 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:08,532 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689121958531 2023-07-12 00:32:08,532 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 00:32:08,532 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 00:32:08,532 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 00:32:08,533 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 00:32:08,533 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 00:32:08,533 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 00:32:08,533 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:08,533 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 00:32:08,533 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-12 00:32:08,535 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 00:32:08,535 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 00:32:08,536 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 00:32:08,536 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 00:32:08,536 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 00:32:08,536 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 00:32:08,539 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689121928536,5,FailOnTimeoutGroup] 2023-07-12 00:32:08,541 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689121928540,5,FailOnTimeoutGroup] 2023-07-12 00:32:08,541 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:08,542 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 00:32:08,542 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:08,542 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:08,551 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 00:32:08,552 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 00:32:08,552 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983 2023-07-12 00:32:08,560 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:08,561 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 00:32:08,563 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/info 2023-07-12 00:32:08,564 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 00:32:08,564 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:08,564 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 00:32:08,566 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/rep_barrier 2023-07-12 00:32:08,566 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 00:32:08,567 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:08,567 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 00:32:08,569 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/table 2023-07-12 00:32:08,569 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 00:32:08,570 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:08,571 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740 2023-07-12 00:32:08,571 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740 2023-07-12 00:32:08,573 INFO [RS:0;jenkins-hbase20:42069] regionserver.HRegionServer(951): ClusterId : b855a1de-f551-45f5-8b9b-f7b8ff57b75f 2023-07-12 00:32:08,573 DEBUG [RS:0;jenkins-hbase20:42069] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 00:32:08,574 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 00:32:08,574 INFO [RS:1;jenkins-hbase20:33475] regionserver.HRegionServer(951): ClusterId : b855a1de-f551-45f5-8b9b-f7b8ff57b75f 2023-07-12 00:32:08,574 DEBUG [RS:1;jenkins-hbase20:33475] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 00:32:08,575 DEBUG [RS:0;jenkins-hbase20:42069] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 00:32:08,575 DEBUG [RS:0;jenkins-hbase20:42069] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 00:32:08,575 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 00:32:08,575 DEBUG [RS:1;jenkins-hbase20:33475] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 00:32:08,576 DEBUG [RS:1;jenkins-hbase20:33475] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 00:32:08,577 DEBUG [RS:0;jenkins-hbase20:42069] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 00:32:08,577 DEBUG [RS:1;jenkins-hbase20:33475] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 00:32:08,580 DEBUG [RS:0;jenkins-hbase20:42069] zookeeper.ReadOnlyZKClient(139): Connect 0x176c4c2d to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:32:08,584 DEBUG [RS:1;jenkins-hbase20:33475] zookeeper.ReadOnlyZKClient(139): Connect 0x01a1e03e to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:32:08,590 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:08,592 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69143156, jitterRate=0.030313313007354736}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 00:32:08,592 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 00:32:08,592 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 00:32:08,592 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 00:32:08,593 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 00:32:08,593 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 00:32:08,593 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 00:32:08,593 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 00:32:08,593 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 00:32:08,594 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 00:32:08,594 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-12 00:32:08,594 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-12 00:32:08,596 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-12 00:32:08,598 DEBUG [RS:0;jenkins-hbase20:42069] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@19aba0b0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:32:08,598 DEBUG [RS:0;jenkins-hbase20:42069] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d94275d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-12 00:32:08,598 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-12 00:32:08,600 DEBUG [RS:1;jenkins-hbase20:33475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f660e7a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:32:08,601 DEBUG [RS:1;jenkins-hbase20:33475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b598b4e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-12 00:32:08,609 DEBUG [RS:0;jenkins-hbase20:42069] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:42069 2023-07-12 00:32:08,609 INFO [RS:0;jenkins-hbase20:42069] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 00:32:08,609 INFO [RS:0;jenkins-hbase20:42069] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 00:32:08,609 DEBUG [RS:0;jenkins-hbase20:42069] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 00:32:08,610 INFO [RS:0;jenkins-hbase20:42069] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase20.apache.org,42265,1689121928263 with isa=jenkins-hbase20.apache.org/148.251.75.209:42069, startcode=1689121928319 2023-07-12 00:32:08,610 DEBUG [RS:0;jenkins-hbase20:42069] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 00:32:08,610 DEBUG [RS:1;jenkins-hbase20:33475] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase20:33475 2023-07-12 00:32:08,610 INFO [RS:1;jenkins-hbase20:33475] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 00:32:08,610 INFO [RS:1;jenkins-hbase20:33475] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 00:32:08,611 DEBUG [RS:1;jenkins-hbase20:33475] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 00:32:08,611 INFO [RS:1;jenkins-hbase20:33475] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase20.apache.org,42265,1689121928263 with isa=jenkins-hbase20.apache.org/148.251.75.209:33475, startcode=1689121928349 2023-07-12 00:32:08,611 DEBUG [RS:1;jenkins-hbase20:33475] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 00:32:08,613 INFO [RS-EventLoopGroup-15-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36029, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.6 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 00:32:08,615 INFO [RS-EventLoopGroup-15-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39715, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.7 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 00:32:08,615 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42265] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:08,616 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42265] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,33475,1689121928349 2023-07-12 00:32:08,616 DEBUG [RS:0;jenkins-hbase20:42069] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983 2023-07-12 00:32:08,616 DEBUG [RS:0;jenkins-hbase20:42069] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:44295 2023-07-12 00:32:08,616 DEBUG [RS:0;jenkins-hbase20:42069] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 00:32:08,617 DEBUG [RS:1;jenkins-hbase20:33475] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983 2023-07-12 00:32:08,617 DEBUG [RS:1;jenkins-hbase20:33475] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:44295 2023-07-12 00:32:08,617 DEBUG [RS:1;jenkins-hbase20:33475] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 00:32:08,617 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:08,619 DEBUG [RS:0;jenkins-hbase20:42069] zookeeper.ZKUtil(162): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:08,619 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,33475,1689121928349] 2023-07-12 00:32:08,619 WARN [RS:0;jenkins-hbase20:42069] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 00:32:08,619 DEBUG [RS:1;jenkins-hbase20:33475] zookeeper.ZKUtil(162): regionserver:33475-0x1007e9741810021, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,33475,1689121928349 2023-07-12 00:32:08,619 INFO [RS:0;jenkins-hbase20:42069] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:32:08,619 WARN [RS:1;jenkins-hbase20:33475] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 00:32:08,619 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,42069,1689121928319] 2023-07-12 00:32:08,619 DEBUG [RS:0;jenkins-hbase20:42069] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:08,619 INFO [RS:1;jenkins-hbase20:33475] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:32:08,619 DEBUG [RS:1;jenkins-hbase20:33475] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349 2023-07-12 00:32:08,628 DEBUG [RS:1;jenkins-hbase20:33475] zookeeper.ZKUtil(162): regionserver:33475-0x1007e9741810021, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,33475,1689121928349 2023-07-12 00:32:08,628 DEBUG [RS:0;jenkins-hbase20:42069] zookeeper.ZKUtil(162): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,33475,1689121928349 2023-07-12 00:32:08,628 DEBUG [RS:1;jenkins-hbase20:33475] zookeeper.ZKUtil(162): regionserver:33475-0x1007e9741810021, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:08,628 DEBUG [RS:0;jenkins-hbase20:42069] zookeeper.ZKUtil(162): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:08,629 DEBUG [RS:1;jenkins-hbase20:33475] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 00:32:08,629 INFO [RS:1;jenkins-hbase20:33475] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 00:32:08,629 DEBUG [RS:0;jenkins-hbase20:42069] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 00:32:08,630 INFO [RS:0;jenkins-hbase20:42069] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 00:32:08,631 INFO [RS:0;jenkins-hbase20:42069] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 00:32:08,631 INFO [RS:1;jenkins-hbase20:33475] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 00:32:08,632 INFO [RS:0;jenkins-hbase20:42069] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 00:32:08,632 INFO [RS:0;jenkins-hbase20:42069] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:08,632 INFO [RS:1;jenkins-hbase20:33475] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 00:32:08,632 INFO [RS:1;jenkins-hbase20:33475] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:08,633 INFO [RS:0;jenkins-hbase20:42069] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 00:32:08,634 INFO [RS:1;jenkins-hbase20:33475] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 00:32:08,637 INFO [RS:0;jenkins-hbase20:42069] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:08,637 INFO [RS:1;jenkins-hbase20:33475] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:08,637 DEBUG [RS:0;jenkins-hbase20:42069] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:08,637 DEBUG [RS:1;jenkins-hbase20:33475] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:08,637 DEBUG [RS:0;jenkins-hbase20:42069] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:08,637 DEBUG [RS:1;jenkins-hbase20:33475] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:08,637 DEBUG [RS:0;jenkins-hbase20:42069] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:08,637 DEBUG [RS:1;jenkins-hbase20:33475] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:08,637 DEBUG [RS:0;jenkins-hbase20:42069] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:08,637 DEBUG [RS:1;jenkins-hbase20:33475] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:08,637 DEBUG [RS:0;jenkins-hbase20:42069] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:08,637 DEBUG [RS:1;jenkins-hbase20:33475] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:08,637 DEBUG [RS:0;jenkins-hbase20:42069] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-12 00:32:08,637 DEBUG [RS:1;jenkins-hbase20:33475] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-12 00:32:08,637 DEBUG [RS:0;jenkins-hbase20:42069] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:08,637 DEBUG [RS:1;jenkins-hbase20:33475] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:08,637 DEBUG [RS:0;jenkins-hbase20:42069] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:08,637 DEBUG [RS:1;jenkins-hbase20:33475] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:08,637 DEBUG [RS:0;jenkins-hbase20:42069] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:08,637 DEBUG [RS:1;jenkins-hbase20:33475] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:08,637 DEBUG [RS:0;jenkins-hbase20:42069] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:08,638 DEBUG [RS:1;jenkins-hbase20:33475] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:08,640 INFO [RS:0;jenkins-hbase20:42069] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:08,640 INFO [RS:0;jenkins-hbase20:42069] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:08,640 INFO [RS:0;jenkins-hbase20:42069] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:08,641 INFO [RS:1;jenkins-hbase20:33475] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:08,641 INFO [RS:1;jenkins-hbase20:33475] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:08,641 INFO [RS:1;jenkins-hbase20:33475] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:08,653 INFO [RS:1;jenkins-hbase20:33475] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 00:32:08,653 INFO [RS:1;jenkins-hbase20:33475] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,33475,1689121928349-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:08,655 INFO [RS:0;jenkins-hbase20:42069] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 00:32:08,655 INFO [RS:0;jenkins-hbase20:42069] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,42069,1689121928319-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:08,665 INFO [RS:1;jenkins-hbase20:33475] regionserver.Replication(203): jenkins-hbase20.apache.org,33475,1689121928349 started 2023-07-12 00:32:08,666 INFO [RS:1;jenkins-hbase20:33475] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,33475,1689121928349, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:33475, sessionid=0x1007e9741810021 2023-07-12 00:32:08,666 DEBUG [RS:1;jenkins-hbase20:33475] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 00:32:08,666 DEBUG [RS:1;jenkins-hbase20:33475] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,33475,1689121928349 2023-07-12 00:32:08,666 DEBUG [RS:1;jenkins-hbase20:33475] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,33475,1689121928349' 2023-07-12 00:32:08,666 DEBUG [RS:1;jenkins-hbase20:33475] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 00:32:08,666 DEBUG [RS:1;jenkins-hbase20:33475] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 00:32:08,666 DEBUG [RS:1;jenkins-hbase20:33475] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 00:32:08,666 DEBUG [RS:1;jenkins-hbase20:33475] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 00:32:08,666 DEBUG [RS:1;jenkins-hbase20:33475] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,33475,1689121928349 2023-07-12 00:32:08,666 DEBUG [RS:1;jenkins-hbase20:33475] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,33475,1689121928349' 2023-07-12 00:32:08,666 DEBUG [RS:1;jenkins-hbase20:33475] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 00:32:08,667 DEBUG [RS:1;jenkins-hbase20:33475] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 00:32:08,667 DEBUG [RS:1;jenkins-hbase20:33475] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 00:32:08,667 INFO [RS:1;jenkins-hbase20:33475] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 00:32:08,667 INFO [RS:1;jenkins-hbase20:33475] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 00:32:08,669 INFO [RS:0;jenkins-hbase20:42069] regionserver.Replication(203): jenkins-hbase20.apache.org,42069,1689121928319 started 2023-07-12 00:32:08,669 INFO [RS:0;jenkins-hbase20:42069] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,42069,1689121928319, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:42069, sessionid=0x1007e9741810020 2023-07-12 00:32:08,672 DEBUG [RS:0;jenkins-hbase20:42069] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 00:32:08,672 DEBUG [RS:0;jenkins-hbase20:42069] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:08,672 DEBUG [RS:0;jenkins-hbase20:42069] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,42069,1689121928319' 2023-07-12 00:32:08,672 DEBUG [RS:0;jenkins-hbase20:42069] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 00:32:08,673 DEBUG [RS:0;jenkins-hbase20:42069] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 00:32:08,673 DEBUG [RS:0;jenkins-hbase20:42069] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 00:32:08,673 DEBUG [RS:0;jenkins-hbase20:42069] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 00:32:08,673 DEBUG [RS:0;jenkins-hbase20:42069] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:08,673 DEBUG [RS:0;jenkins-hbase20:42069] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,42069,1689121928319' 2023-07-12 00:32:08,673 DEBUG [RS:0;jenkins-hbase20:42069] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 00:32:08,674 DEBUG [RS:0;jenkins-hbase20:42069] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 00:32:08,674 DEBUG [RS:0;jenkins-hbase20:42069] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 00:32:08,674 INFO [RS:0;jenkins-hbase20:42069] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 00:32:08,674 INFO [RS:0;jenkins-hbase20:42069] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 00:32:08,748 DEBUG [jenkins-hbase20:42265] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-12 00:32:08,748 DEBUG [jenkins-hbase20:42265] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-07-12 00:32:08,751 DEBUG [jenkins-hbase20:42265] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 00:32:08,751 DEBUG [jenkins-hbase20:42265] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 00:32:08,751 DEBUG [jenkins-hbase20:42265] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 00:32:08,752 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,33475,1689121928349, state=OPENING 2023-07-12 00:32:08,753 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-12 00:32:08,754 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:08,754 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:08,754 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 00:32:08,754 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 00:32:08,754 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,33475,1689121928349}] 2023-07-12 00:32:08,769 INFO [RS:1;jenkins-hbase20:33475] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C33475%2C1689121928349, suffix=, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/oldWALs, maxLogs=32 2023-07-12 00:32:08,776 INFO [RS:0;jenkins-hbase20:42069] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C42069%2C1689121928319, suffix=, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,42069,1689121928319, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/oldWALs, maxLogs=32 2023-07-12 00:32:08,786 DEBUG [RS-EventLoopGroup-18-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:32:08,786 DEBUG [RS-EventLoopGroup-18-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:32:08,788 INFO [RS:1;jenkins-hbase20:33475] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349/jenkins-hbase20.apache.org%2C33475%2C1689121928349.1689121928769 2023-07-12 00:32:08,789 DEBUG [RS:1;jenkins-hbase20:33475] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK], DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK]] 2023-07-12 00:32:08,802 DEBUG [RS-EventLoopGroup-18-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:32:08,803 DEBUG [RS-EventLoopGroup-18-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:32:08,804 INFO [RS:0;jenkins-hbase20:42069] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,42069,1689121928319/jenkins-hbase20.apache.org%2C42069%2C1689121928319.1689121928777 2023-07-12 00:32:08,806 DEBUG [RS:0;jenkins-hbase20:42069] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK], DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK]] 2023-07-12 00:32:08,910 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,33475,1689121928349 2023-07-12 00:32:08,910 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 00:32:08,911 INFO [RS-EventLoopGroup-18-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49188, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 00:32:08,918 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-12 00:32:08,919 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:32:08,921 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C33475%2C1689121928349.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/oldWALs, maxLogs=32 2023-07-12 00:32:08,935 DEBUG [RS-EventLoopGroup-18-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:32:08,935 DEBUG [RS-EventLoopGroup-18-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:32:08,937 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349/jenkins-hbase20.apache.org%2C33475%2C1689121928349.meta.1689121928922.meta 2023-07-12 00:32:08,937 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK], DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK]] 2023-07-12 00:32:08,937 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-12 00:32:08,937 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-12 00:32:08,938 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-12 00:32:08,938 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-12 00:32:08,938 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-12 00:32:08,938 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:08,938 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-12 00:32:08,938 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-12 00:32:08,941 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 00:32:08,943 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/info 2023-07-12 00:32:08,943 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/info 2023-07-12 00:32:08,943 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 00:32:08,944 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:08,944 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 00:32:08,945 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/rep_barrier 2023-07-12 00:32:08,946 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/rep_barrier 2023-07-12 00:32:08,946 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 00:32:08,946 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:08,947 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 00:32:08,948 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/table 2023-07-12 00:32:08,948 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/table 2023-07-12 00:32:08,948 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 00:32:08,949 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:08,950 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740 2023-07-12 00:32:08,951 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740 2023-07-12 00:32:08,953 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 00:32:08,955 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 00:32:08,955 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72505702, jitterRate=0.0804191529750824}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 00:32:08,955 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 00:32:08,956 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689121928910 2023-07-12 00:32:08,968 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-12 00:32:08,969 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-12 00:32:08,970 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,33475,1689121928349, state=OPEN 2023-07-12 00:32:08,971 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 00:32:08,971 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 00:32:08,971 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 00:32:08,971 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 00:32:08,974 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-12 00:32:08,974 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,33475,1689121928349 in 217 msec 2023-07-12 00:32:08,979 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-12 00:32:08,980 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 380 msec 2023-07-12 00:32:08,982 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 459 msec 2023-07-12 00:32:08,982 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689121928982, completionTime=-1 2023-07-12 00:32:08,982 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-12 00:32:08,982 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 00:32:08,985 DEBUG [hconnection-0x1cd3695a-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 00:32:08,987 INFO [RS-EventLoopGroup-18-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49192, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 00:32:08,988 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-12 00:32:08,989 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689121988989 2023-07-12 00:32:08,989 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689122048989 2023-07-12 00:32:08,989 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 6 msec 2023-07-12 00:32:08,997 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 00:32:09,001 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,42265,1689121928263-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:09,001 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,42265,1689121928263-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:09,001 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,42265,1689121928263-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:09,001 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:42265, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:09,001 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:09,001 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-12 00:32:09,002 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 00:32:09,003 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-12 00:32:09,013 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 00:32:09,016 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 00:32:09,022 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 00:32:09,026 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/hbase/namespace/96bd68d39166bf0c53ccc701580614f1 2023-07-12 00:32:09,028 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/hbase/namespace/96bd68d39166bf0c53ccc701580614f1 empty. 2023-07-12 00:32:09,029 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/hbase/namespace/96bd68d39166bf0c53ccc701580614f1 2023-07-12 00:32:09,029 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-12 00:32:09,042 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-12 00:32:09,045 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 96bd68d39166bf0c53ccc701580614f1, NAME => 'hbase:namespace,,1689121929001.96bd68d39166bf0c53ccc701580614f1.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp 2023-07-12 00:32:09,099 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689121929001.96bd68d39166bf0c53ccc701580614f1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:09,099 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 96bd68d39166bf0c53ccc701580614f1, disabling compactions & flushes 2023-07-12 00:32:09,099 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689121929001.96bd68d39166bf0c53ccc701580614f1. 2023-07-12 00:32:09,099 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689121929001.96bd68d39166bf0c53ccc701580614f1. 2023-07-12 00:32:09,099 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689121929001.96bd68d39166bf0c53ccc701580614f1. after waiting 0 ms 2023-07-12 00:32:09,099 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689121929001.96bd68d39166bf0c53ccc701580614f1. 2023-07-12 00:32:09,100 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689121929001.96bd68d39166bf0c53ccc701580614f1. 2023-07-12 00:32:09,100 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 96bd68d39166bf0c53ccc701580614f1: 2023-07-12 00:32:09,103 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 00:32:09,104 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689121929001.96bd68d39166bf0c53ccc701580614f1.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689121929104"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689121929104"}]},"ts":"1689121929104"} 2023-07-12 00:32:09,107 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 00:32:09,109 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 00:32:09,109 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689121929109"}]},"ts":"1689121929109"} 2023-07-12 00:32:09,111 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-12 00:32:09,113 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-07-12 00:32:09,114 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 00:32:09,114 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 00:32:09,114 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 00:32:09,114 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=96bd68d39166bf0c53ccc701580614f1, ASSIGN}] 2023-07-12 00:32:09,117 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=96bd68d39166bf0c53ccc701580614f1, ASSIGN 2023-07-12 00:32:09,118 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=96bd68d39166bf0c53ccc701580614f1, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,42069,1689121928319; forceNewPlan=false, retain=false 2023-07-12 00:32:09,268 INFO [jenkins-hbase20:42265] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-12 00:32:09,269 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=96bd68d39166bf0c53ccc701580614f1, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:09,269 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689121929001.96bd68d39166bf0c53ccc701580614f1.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689121929269"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121929269"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121929269"}]},"ts":"1689121929269"} 2023-07-12 00:32:09,278 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 96bd68d39166bf0c53ccc701580614f1, server=jenkins-hbase20.apache.org,42069,1689121928319}] 2023-07-12 00:32:09,433 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:09,433 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 00:32:09,434 INFO [RS-EventLoopGroup-17-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48446, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 00:32:09,438 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689121929001.96bd68d39166bf0c53ccc701580614f1. 2023-07-12 00:32:09,438 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 96bd68d39166bf0c53ccc701580614f1, NAME => 'hbase:namespace,,1689121929001.96bd68d39166bf0c53ccc701580614f1.', STARTKEY => '', ENDKEY => ''} 2023-07-12 00:32:09,438 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 96bd68d39166bf0c53ccc701580614f1 2023-07-12 00:32:09,438 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689121929001.96bd68d39166bf0c53ccc701580614f1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:09,438 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 96bd68d39166bf0c53ccc701580614f1 2023-07-12 00:32:09,438 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 96bd68d39166bf0c53ccc701580614f1 2023-07-12 00:32:09,440 INFO [StoreOpener-96bd68d39166bf0c53ccc701580614f1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 96bd68d39166bf0c53ccc701580614f1 2023-07-12 00:32:09,441 DEBUG [StoreOpener-96bd68d39166bf0c53ccc701580614f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/96bd68d39166bf0c53ccc701580614f1/info 2023-07-12 00:32:09,441 DEBUG [StoreOpener-96bd68d39166bf0c53ccc701580614f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/96bd68d39166bf0c53ccc701580614f1/info 2023-07-12 00:32:09,442 INFO [StoreOpener-96bd68d39166bf0c53ccc701580614f1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 96bd68d39166bf0c53ccc701580614f1 columnFamilyName info 2023-07-12 00:32:09,442 INFO [StoreOpener-96bd68d39166bf0c53ccc701580614f1-1] regionserver.HStore(310): Store=96bd68d39166bf0c53ccc701580614f1/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:09,443 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/96bd68d39166bf0c53ccc701580614f1 2023-07-12 00:32:09,444 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/96bd68d39166bf0c53ccc701580614f1 2023-07-12 00:32:09,446 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 96bd68d39166bf0c53ccc701580614f1 2023-07-12 00:32:09,449 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/96bd68d39166bf0c53ccc701580614f1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:09,449 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 96bd68d39166bf0c53ccc701580614f1; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67175650, jitterRate=9.951889514923096E-4}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:09,449 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 96bd68d39166bf0c53ccc701580614f1: 2023-07-12 00:32:09,450 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689121929001.96bd68d39166bf0c53ccc701580614f1., pid=6, masterSystemTime=1689121929433 2023-07-12 00:32:09,454 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689121929001.96bd68d39166bf0c53ccc701580614f1. 2023-07-12 00:32:09,455 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689121929001.96bd68d39166bf0c53ccc701580614f1. 2023-07-12 00:32:09,455 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=96bd68d39166bf0c53ccc701580614f1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:09,456 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689121929001.96bd68d39166bf0c53ccc701580614f1.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689121929455"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121929455"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121929455"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121929455"}]},"ts":"1689121929455"} 2023-07-12 00:32:09,459 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-12 00:32:09,459 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 96bd68d39166bf0c53ccc701580614f1, server=jenkins-hbase20.apache.org,42069,1689121928319 in 179 msec 2023-07-12 00:32:09,461 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-12 00:32:09,461 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=96bd68d39166bf0c53ccc701580614f1, ASSIGN in 345 msec 2023-07-12 00:32:09,462 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 00:32:09,462 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689121929462"}]},"ts":"1689121929462"} 2023-07-12 00:32:09,464 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-12 00:32:09,466 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 00:32:09,467 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 464 msec 2023-07-12 00:32:09,505 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-12 00:32:09,506 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-12 00:32:09,506 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:09,506 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:09,511 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 00:32:09,513 INFO [RS-EventLoopGroup-17-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48456, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 00:32:09,516 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-12 00:32:09,530 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 00:32:09,534 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 17 msec 2023-07-12 00:32:09,539 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-12 00:32:09,547 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 00:32:09,550 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 11 msec 2023-07-12 00:32:09,563 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 00:32:09,564 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 00:32:09,564 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 1.179sec 2023-07-12 00:32:09,565 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 00:32:09,565 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 00:32:09,565 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 00:32:09,565 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,42265,1689121928263-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 00:32:09,565 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,42265,1689121928263-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 00:32:09,567 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 00:32:09,586 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ReadOnlyZKClient(139): Connect 0x0d42fd21 to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:32:09,591 DEBUG [Listener at localhost.localdomain/34481] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4273573, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:32:09,593 DEBUG [hconnection-0x3fc558ca-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 00:32:09,595 INFO [RS-EventLoopGroup-18-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49206, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 00:32:09,597 INFO [Listener at localhost.localdomain/34481] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,42265,1689121928263 2023-07-12 00:32:09,611 INFO [Listener at localhost.localdomain/34481] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-07-12 00:32:09,611 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:09,611 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:09,611 INFO [Listener at localhost.localdomain/34481] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 00:32:09,611 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:09,611 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 00:32:09,611 INFO [Listener at localhost.localdomain/34481] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 00:32:09,612 INFO [Listener at localhost.localdomain/34481] ipc.NettyRpcServer(120): Bind to /148.251.75.209:37887 2023-07-12 00:32:09,612 INFO [Listener at localhost.localdomain/34481] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 00:32:09,616 DEBUG [Listener at localhost.localdomain/34481] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 00:32:09,616 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:09,617 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:09,619 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:37887 connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:32:09,622 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:378870x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:32:09,624 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): regionserver:378870x0, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:32:09,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:37887-0x1007e9741810026 connected 2023-07-12 00:32:09,628 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): regionserver:37887-0x1007e9741810026, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-12 00:32:09,629 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:37887-0x1007e9741810026, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 00:32:09,633 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37887 2023-07-12 00:32:09,633 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37887 2023-07-12 00:32:09,633 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37887 2023-07-12 00:32:09,635 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37887 2023-07-12 00:32:09,635 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37887 2023-07-12 00:32:09,640 INFO [RS:2;jenkins-hbase20:37887] regionserver.HRegionServer(951): ClusterId : b855a1de-f551-45f5-8b9b-f7b8ff57b75f 2023-07-12 00:32:09,640 DEBUG [RS:2;jenkins-hbase20:37887] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 00:32:09,642 DEBUG [RS:2;jenkins-hbase20:37887] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 00:32:09,642 DEBUG [RS:2;jenkins-hbase20:37887] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 00:32:09,643 DEBUG [RS:2;jenkins-hbase20:37887] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 00:32:09,646 DEBUG [RS:2;jenkins-hbase20:37887] zookeeper.ReadOnlyZKClient(139): Connect 0x4caa4cc4 to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:32:09,656 DEBUG [RS:2;jenkins-hbase20:37887] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@426dbb21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:32:09,656 DEBUG [RS:2;jenkins-hbase20:37887] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@465227b4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-12 00:32:09,664 DEBUG [RS:2;jenkins-hbase20:37887] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase20:37887 2023-07-12 00:32:09,664 INFO [RS:2;jenkins-hbase20:37887] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 00:32:09,664 INFO [RS:2;jenkins-hbase20:37887] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 00:32:09,664 DEBUG [RS:2;jenkins-hbase20:37887] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 00:32:09,665 INFO [RS:2;jenkins-hbase20:37887] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase20.apache.org,42265,1689121928263 with isa=jenkins-hbase20.apache.org/148.251.75.209:37887, startcode=1689121929610 2023-07-12 00:32:09,665 DEBUG [RS:2;jenkins-hbase20:37887] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 00:32:09,667 INFO [RS-EventLoopGroup-15-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:58243, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.8 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 00:32:09,668 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42265] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,37887,1689121929610 2023-07-12 00:32:09,668 DEBUG [RS:2;jenkins-hbase20:37887] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983 2023-07-12 00:32:09,668 DEBUG [RS:2;jenkins-hbase20:37887] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:44295 2023-07-12 00:32:09,668 DEBUG [RS:2;jenkins-hbase20:37887] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 00:32:09,669 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:33475-0x1007e9741810021, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:09,669 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:09,669 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:09,670 DEBUG [RS:2;jenkins-hbase20:37887] zookeeper.ZKUtil(162): regionserver:37887-0x1007e9741810026, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,37887,1689121929610 2023-07-12 00:32:09,670 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,37887,1689121929610] 2023-07-12 00:32:09,670 WARN [RS:2;jenkins-hbase20:37887] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 00:32:09,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33475-0x1007e9741810021, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,37887,1689121929610 2023-07-12 00:32:09,670 INFO [RS:2;jenkins-hbase20:37887] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:32:09,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,37887,1689121929610 2023-07-12 00:32:09,670 DEBUG [RS:2;jenkins-hbase20:37887] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37887,1689121929610 2023-07-12 00:32:09,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33475-0x1007e9741810021, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,33475,1689121928349 2023-07-12 00:32:09,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,33475,1689121928349 2023-07-12 00:32:09,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33475-0x1007e9741810021, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:09,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:09,675 DEBUG [RS:2;jenkins-hbase20:37887] zookeeper.ZKUtil(162): regionserver:37887-0x1007e9741810026, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,37887,1689121929610 2023-07-12 00:32:09,676 DEBUG [RS:2;jenkins-hbase20:37887] zookeeper.ZKUtil(162): regionserver:37887-0x1007e9741810026, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,33475,1689121928349 2023-07-12 00:32:09,676 DEBUG [RS:2;jenkins-hbase20:37887] zookeeper.ZKUtil(162): regionserver:37887-0x1007e9741810026, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:09,677 DEBUG [RS:2;jenkins-hbase20:37887] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 00:32:09,677 INFO [RS:2;jenkins-hbase20:37887] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 00:32:09,679 INFO [RS:2;jenkins-hbase20:37887] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 00:32:09,679 INFO [RS:2;jenkins-hbase20:37887] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 00:32:09,679 INFO [RS:2;jenkins-hbase20:37887] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:09,679 INFO [RS:2;jenkins-hbase20:37887] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 00:32:09,681 INFO [RS:2;jenkins-hbase20:37887] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:09,681 DEBUG [RS:2;jenkins-hbase20:37887] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:09,681 DEBUG [RS:2;jenkins-hbase20:37887] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:09,681 DEBUG [RS:2;jenkins-hbase20:37887] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:09,681 DEBUG [RS:2;jenkins-hbase20:37887] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:09,681 DEBUG [RS:2;jenkins-hbase20:37887] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:09,681 DEBUG [RS:2;jenkins-hbase20:37887] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-12 00:32:09,681 DEBUG [RS:2;jenkins-hbase20:37887] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:09,681 DEBUG [RS:2;jenkins-hbase20:37887] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:09,681 DEBUG [RS:2;jenkins-hbase20:37887] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:09,681 DEBUG [RS:2;jenkins-hbase20:37887] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:09,682 INFO [RS:2;jenkins-hbase20:37887] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:09,683 INFO [RS:2;jenkins-hbase20:37887] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:09,683 INFO [RS:2;jenkins-hbase20:37887] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:09,695 INFO [RS:2;jenkins-hbase20:37887] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 00:32:09,695 INFO [RS:2;jenkins-hbase20:37887] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,37887,1689121929610-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:09,707 INFO [RS:2;jenkins-hbase20:37887] regionserver.Replication(203): jenkins-hbase20.apache.org,37887,1689121929610 started 2023-07-12 00:32:09,707 INFO [RS:2;jenkins-hbase20:37887] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,37887,1689121929610, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:37887, sessionid=0x1007e9741810026 2023-07-12 00:32:09,707 DEBUG [RS:2;jenkins-hbase20:37887] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 00:32:09,707 DEBUG [RS:2;jenkins-hbase20:37887] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,37887,1689121929610 2023-07-12 00:32:09,707 DEBUG [RS:2;jenkins-hbase20:37887] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,37887,1689121929610' 2023-07-12 00:32:09,707 DEBUG [RS:2;jenkins-hbase20:37887] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 00:32:09,707 DEBUG [RS:2;jenkins-hbase20:37887] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 00:32:09,710 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-12 00:32:09,710 DEBUG [RS:2;jenkins-hbase20:37887] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 00:32:09,710 DEBUG [RS:2;jenkins-hbase20:37887] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 00:32:09,711 DEBUG [RS:2;jenkins-hbase20:37887] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,37887,1689121929610 2023-07-12 00:32:09,711 DEBUG [RS:2;jenkins-hbase20:37887] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,37887,1689121929610' 2023-07-12 00:32:09,711 DEBUG [RS:2;jenkins-hbase20:37887] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 00:32:09,711 DEBUG [RS:2;jenkins-hbase20:37887] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 00:32:09,711 DEBUG [RS:2;jenkins-hbase20:37887] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 00:32:09,711 INFO [RS:2;jenkins-hbase20:37887] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 00:32:09,711 INFO [RS:2;jenkins-hbase20:37887] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 00:32:09,712 INFO [RS-EventLoopGroup-15-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59716, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-12 00:32:09,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42265] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'testLogSplittingAfterMasterRecoveryDueToZKExpiry', {NAME => 'col', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 00:32:09,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42265] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry 2023-07-12 00:32:09,719 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 00:32:09,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42265] master.MasterRpcServices(700): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "testLogSplittingAfterMasterRecoveryDueToZKExpiry" procId is: 9 2023-07-12 00:32:09,720 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 00:32:09,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42265] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 00:32:09,726 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/977900e03dd851ff67f68c586e17ff0e 2023-07-12 00:32:09,726 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39 2023-07-12 00:32:09,726 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/38818f72237f4c84213ef9820649179c 2023-07-12 00:32:09,726 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/02432dc8a0cc182e04b7721d9cac70d0 2023-07-12 00:32:09,726 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/e05a790448a127b53382088bce1a74d6 2023-07-12 00:32:09,726 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/810f496a1e86b8d9ebefb6030254ac4d 2023-07-12 00:32:09,727 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/977900e03dd851ff67f68c586e17ff0e empty. 2023-07-12 00:32:09,727 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/38818f72237f4c84213ef9820649179c empty. 2023-07-12 00:32:09,727 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/e05a790448a127b53382088bce1a74d6 empty. 2023-07-12 00:32:09,727 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39 empty. 2023-07-12 00:32:09,727 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/977900e03dd851ff67f68c586e17ff0e 2023-07-12 00:32:09,728 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/02432dc8a0cc182e04b7721d9cac70d0 empty. 2023-07-12 00:32:09,728 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/810f496a1e86b8d9ebefb6030254ac4d empty. 2023-07-12 00:32:09,728 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/e05a790448a127b53382088bce1a74d6 2023-07-12 00:32:09,728 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39 2023-07-12 00:32:09,728 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/38818f72237f4c84213ef9820649179c 2023-07-12 00:32:09,728 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/02432dc8a0cc182e04b7721d9cac70d0 2023-07-12 00:32:09,728 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/810f496a1e86b8d9ebefb6030254ac4d 2023-07-12 00:32:09,728 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testLogSplittingAfterMasterRecoveryDueToZKExpiry regions 2023-07-12 00:32:09,740 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/.tabledesc/.tableinfo.0000000001 2023-07-12 00:32:09,741 INFO [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(7675): creating {ENCODED => 2cba8ed741e888a6ec6550e0722fbc39, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testLogSplittingAfterMasterRecoveryDueToZKExpiry', {NAME => 'col', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp 2023-07-12 00:32:09,743 INFO [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(7675): creating {ENCODED => 977900e03dd851ff67f68c586e17ff0e, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1689121929714.977900e03dd851ff67f68c586e17ff0e.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testLogSplittingAfterMasterRecoveryDueToZKExpiry', {NAME => 'col', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp 2023-07-12 00:32:09,743 INFO [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(7675): creating {ENCODED => 810f496a1e86b8d9ebefb6030254ac4d, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1689121929714.810f496a1e86b8d9ebefb6030254ac4d.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testLogSplittingAfterMasterRecoveryDueToZKExpiry', {NAME => 'col', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp 2023-07-12 00:32:09,814 INFO [RS:2;jenkins-hbase20:37887] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C37887%2C1689121929610, suffix=, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37887,1689121929610, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/oldWALs, maxLogs=32 2023-07-12 00:32:09,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42265] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 00:32:09,830 DEBUG [RS-EventLoopGroup-19-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:32:09,830 DEBUG [RS-EventLoopGroup-19-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:32:09,832 INFO [RS:2;jenkins-hbase20:37887] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,37887,1689121929610/jenkins-hbase20.apache.org%2C37887%2C1689121929610.1689121929814 2023-07-12 00:32:09,834 DEBUG [RS:2;jenkins-hbase20:37887] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK], DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK]] 2023-07-12 00:32:10,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42265] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 00:32:10,179 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(866): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:10,179 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1604): Closing 2cba8ed741e888a6ec6550e0722fbc39, disabling compactions & flushes 2023-07-12 00:32:10,179 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(866): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1689121929714.977900e03dd851ff67f68c586e17ff0e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:10,179 INFO [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1626): Closing region testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39. 2023-07-12 00:32:10,179 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(866): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1689121929714.810f496a1e86b8d9ebefb6030254ac4d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:10,180 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39. 2023-07-12 00:32:10,180 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1604): Closing 810f496a1e86b8d9ebefb6030254ac4d, disabling compactions & flushes 2023-07-12 00:32:10,180 INFO [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1626): Closing region testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1689121929714.810f496a1e86b8d9ebefb6030254ac4d. 2023-07-12 00:32:10,180 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1689121929714.810f496a1e86b8d9ebefb6030254ac4d. 2023-07-12 00:32:10,180 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1714): Acquired close lock on testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1689121929714.810f496a1e86b8d9ebefb6030254ac4d. after waiting 0 ms 2023-07-12 00:32:10,180 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1724): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1689121929714.810f496a1e86b8d9ebefb6030254ac4d. 2023-07-12 00:32:10,180 INFO [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1838): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1689121929714.810f496a1e86b8d9ebefb6030254ac4d. 2023-07-12 00:32:10,180 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1558): Region close journal for 810f496a1e86b8d9ebefb6030254ac4d: 2023-07-12 00:32:10,180 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1714): Acquired close lock on testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39. after waiting 0 ms 2023-07-12 00:32:10,180 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1604): Closing 977900e03dd851ff67f68c586e17ff0e, disabling compactions & flushes 2023-07-12 00:32:10,180 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1724): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39. 2023-07-12 00:32:10,181 INFO [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1838): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39. 2023-07-12 00:32:10,181 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1558): Region close journal for 2cba8ed741e888a6ec6550e0722fbc39: 2023-07-12 00:32:10,181 INFO [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1626): Closing region testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1689121929714.977900e03dd851ff67f68c586e17ff0e. 2023-07-12 00:32:10,181 INFO [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(7675): creating {ENCODED => e05a790448a127b53382088bce1a74d6, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testLogSplittingAfterMasterRecoveryDueToZKExpiry', {NAME => 'col', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp 2023-07-12 00:32:10,181 INFO [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(7675): creating {ENCODED => 02432dc8a0cc182e04b7721d9cac70d0, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1689121929714.02432dc8a0cc182e04b7721d9cac70d0.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testLogSplittingAfterMasterRecoveryDueToZKExpiry', {NAME => 'col', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp 2023-07-12 00:32:10,181 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1689121929714.977900e03dd851ff67f68c586e17ff0e. 2023-07-12 00:32:10,182 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1714): Acquired close lock on testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1689121929714.977900e03dd851ff67f68c586e17ff0e. after waiting 0 ms 2023-07-12 00:32:10,182 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1724): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1689121929714.977900e03dd851ff67f68c586e17ff0e. 2023-07-12 00:32:10,182 INFO [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1838): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1689121929714.977900e03dd851ff67f68c586e17ff0e. 2023-07-12 00:32:10,182 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1558): Region close journal for 977900e03dd851ff67f68c586e17ff0e: 2023-07-12 00:32:10,182 INFO [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(7675): creating {ENCODED => 38818f72237f4c84213ef9820649179c, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1689121929714.38818f72237f4c84213ef9820649179c.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testLogSplittingAfterMasterRecoveryDueToZKExpiry', {NAME => 'col', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp 2023-07-12 00:32:10,210 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(866): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1689121929714.38818f72237f4c84213ef9820649179c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:10,210 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1604): Closing 38818f72237f4c84213ef9820649179c, disabling compactions & flushes 2023-07-12 00:32:10,210 INFO [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1626): Closing region testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1689121929714.38818f72237f4c84213ef9820649179c. 2023-07-12 00:32:10,210 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1689121929714.38818f72237f4c84213ef9820649179c. 2023-07-12 00:32:10,210 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1714): Acquired close lock on testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1689121929714.38818f72237f4c84213ef9820649179c. after waiting 0 ms 2023-07-12 00:32:10,210 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1724): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1689121929714.38818f72237f4c84213ef9820649179c. 2023-07-12 00:32:10,210 INFO [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1838): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1689121929714.38818f72237f4c84213ef9820649179c. 2023-07-12 00:32:10,210 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1558): Region close journal for 38818f72237f4c84213ef9820649179c: 2023-07-12 00:32:10,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42265] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 00:32:10,609 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(866): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:10,609 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(866): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1689121929714.02432dc8a0cc182e04b7721d9cac70d0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:10,609 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1604): Closing e05a790448a127b53382088bce1a74d6, disabling compactions & flushes 2023-07-12 00:32:10,609 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1604): Closing 02432dc8a0cc182e04b7721d9cac70d0, disabling compactions & flushes 2023-07-12 00:32:10,609 INFO [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1626): Closing region testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6. 2023-07-12 00:32:10,610 INFO [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1626): Closing region testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1689121929714.02432dc8a0cc182e04b7721d9cac70d0. 2023-07-12 00:32:10,610 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6. 2023-07-12 00:32:10,610 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1689121929714.02432dc8a0cc182e04b7721d9cac70d0. 2023-07-12 00:32:10,610 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1714): Acquired close lock on testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6. after waiting 0 ms 2023-07-12 00:32:10,610 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1724): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6. 2023-07-12 00:32:10,610 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1714): Acquired close lock on testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1689121929714.02432dc8a0cc182e04b7721d9cac70d0. after waiting 0 ms 2023-07-12 00:32:10,610 INFO [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1838): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6. 2023-07-12 00:32:10,610 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1724): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1689121929714.02432dc8a0cc182e04b7721d9cac70d0. 2023-07-12 00:32:10,610 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1558): Region close journal for e05a790448a127b53382088bce1a74d6: 2023-07-12 00:32:10,610 INFO [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1838): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1689121929714.02432dc8a0cc182e04b7721d9cac70d0. 2023-07-12 00:32:10,610 DEBUG [RegionOpenAndInit-testLogSplittingAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1558): Region close journal for 02432dc8a0cc182e04b7721d9cac70d0: 2023-07-12 00:32:10,613 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 00:32:10,614 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1689121929714.810f496a1e86b8d9ebefb6030254ac4d.","families":{"info":[{"qualifier":"regioninfo","vlen":84,"tag":[],"timestamp":"1689121930614"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689121930614"}]},"ts":"1689121930614"} 2023-07-12 00:32:10,614 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39.","families":{"info":[{"qualifier":"regioninfo","vlen":83,"tag":[],"timestamp":"1689121930614"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689121930614"}]},"ts":"1689121930614"} 2023-07-12 00:32:10,615 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1689121929714.977900e03dd851ff67f68c586e17ff0e.","families":{"info":[{"qualifier":"regioninfo","vlen":84,"tag":[],"timestamp":"1689121930614"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689121930614"}]},"ts":"1689121930614"} 2023-07-12 00:32:10,615 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1689121929714.38818f72237f4c84213ef9820649179c.","families":{"info":[{"qualifier":"regioninfo","vlen":83,"tag":[],"timestamp":"1689121930614"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689121930614"}]},"ts":"1689121930614"} 2023-07-12 00:32:10,615 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6.","families":{"info":[{"qualifier":"regioninfo","vlen":84,"tag":[],"timestamp":"1689121930614"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689121930614"}]},"ts":"1689121930614"} 2023-07-12 00:32:10,615 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1689121929714.02432dc8a0cc182e04b7721d9cac70d0.","families":{"info":[{"qualifier":"regioninfo","vlen":84,"tag":[],"timestamp":"1689121930614"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689121930614"}]},"ts":"1689121930614"} 2023-07-12 00:32:10,659 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-07-12 00:32:10,661 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 00:32:10,661 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689121930661"}]},"ts":"1689121930661"} 2023-07-12 00:32:10,663 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testLogSplittingAfterMasterRecoveryDueToZKExpiry, state=ENABLING in hbase:meta 2023-07-12 00:32:10,665 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-07-12 00:32:10,668 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 00:32:10,668 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 00:32:10,668 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-07-12 00:32:10,668 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 00:32:10,668 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=2cba8ed741e888a6ec6550e0722fbc39, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=977900e03dd851ff67f68c586e17ff0e, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=810f496a1e86b8d9ebefb6030254ac4d, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=02432dc8a0cc182e04b7721d9cac70d0, ASSIGN}, {pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=e05a790448a127b53382088bce1a74d6, ASSIGN}, {pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=38818f72237f4c84213ef9820649179c, ASSIGN}] 2023-07-12 00:32:10,671 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=977900e03dd851ff67f68c586e17ff0e, ASSIGN 2023-07-12 00:32:10,671 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=2cba8ed741e888a6ec6550e0722fbc39, ASSIGN 2023-07-12 00:32:10,671 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=810f496a1e86b8d9ebefb6030254ac4d, ASSIGN 2023-07-12 00:32:10,671 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=02432dc8a0cc182e04b7721d9cac70d0, ASSIGN 2023-07-12 00:32:10,672 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=977900e03dd851ff67f68c586e17ff0e, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,37887,1689121929610; forceNewPlan=false, retain=false 2023-07-12 00:32:10,672 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=2cba8ed741e888a6ec6550e0722fbc39, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,33475,1689121928349; forceNewPlan=false, retain=false 2023-07-12 00:32:10,672 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=02432dc8a0cc182e04b7721d9cac70d0, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,37887,1689121929610; forceNewPlan=false, retain=false 2023-07-12 00:32:10,672 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=38818f72237f4c84213ef9820649179c, ASSIGN 2023-07-12 00:32:10,672 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=810f496a1e86b8d9ebefb6030254ac4d, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,42069,1689121928319; forceNewPlan=false, retain=false 2023-07-12 00:32:10,673 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=e05a790448a127b53382088bce1a74d6, ASSIGN 2023-07-12 00:32:10,673 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=38818f72237f4c84213ef9820649179c, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,42069,1689121928319; forceNewPlan=false, retain=false 2023-07-12 00:32:10,673 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=e05a790448a127b53382088bce1a74d6, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,33475,1689121928349; forceNewPlan=false, retain=false 2023-07-12 00:32:10,822 INFO [jenkins-hbase20:42265] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-07-12 00:32:10,827 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=38818f72237f4c84213ef9820649179c, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:10,827 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=810f496a1e86b8d9ebefb6030254ac4d, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:10,827 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1689121929714.38818f72237f4c84213ef9820649179c.","families":{"info":[{"qualifier":"regioninfo","vlen":83,"tag":[],"timestamp":"1689121930827"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121930827"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121930827"}]},"ts":"1689121930827"} 2023-07-12 00:32:10,827 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=2cba8ed741e888a6ec6550e0722fbc39, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,33475,1689121928349 2023-07-12 00:32:10,827 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=02432dc8a0cc182e04b7721d9cac70d0, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,37887,1689121929610 2023-07-12 00:32:10,827 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1689121929714.810f496a1e86b8d9ebefb6030254ac4d.","families":{"info":[{"qualifier":"regioninfo","vlen":84,"tag":[],"timestamp":"1689121930827"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121930827"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121930827"}]},"ts":"1689121930827"} 2023-07-12 00:32:10,827 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1689121929714.02432dc8a0cc182e04b7721d9cac70d0.","families":{"info":[{"qualifier":"regioninfo","vlen":84,"tag":[],"timestamp":"1689121930827"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121930827"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121930827"}]},"ts":"1689121930827"} 2023-07-12 00:32:10,827 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=977900e03dd851ff67f68c586e17ff0e, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,37887,1689121929610 2023-07-12 00:32:10,828 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1689121929714.977900e03dd851ff67f68c586e17ff0e.","families":{"info":[{"qualifier":"regioninfo","vlen":84,"tag":[],"timestamp":"1689121930827"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121930827"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121930827"}]},"ts":"1689121930827"} 2023-07-12 00:32:10,827 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39.","families":{"info":[{"qualifier":"regioninfo","vlen":83,"tag":[],"timestamp":"1689121930827"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121930827"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121930827"}]},"ts":"1689121930827"} 2023-07-12 00:32:10,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42265] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 00:32:10,829 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE; OpenRegionProcedure 38818f72237f4c84213ef9820649179c, server=jenkins-hbase20.apache.org,42069,1689121928319}] 2023-07-12 00:32:10,831 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=12, state=RUNNABLE; OpenRegionProcedure 810f496a1e86b8d9ebefb6030254ac4d, server=jenkins-hbase20.apache.org,42069,1689121928319}] 2023-07-12 00:32:10,832 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=13, state=RUNNABLE; OpenRegionProcedure 02432dc8a0cc182e04b7721d9cac70d0, server=jenkins-hbase20.apache.org,37887,1689121929610}] 2023-07-12 00:32:10,835 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=e05a790448a127b53382088bce1a74d6, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,33475,1689121928349 2023-07-12 00:32:10,835 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6.","families":{"info":[{"qualifier":"regioninfo","vlen":84,"tag":[],"timestamp":"1689121930835"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121930835"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121930835"}]},"ts":"1689121930835"} 2023-07-12 00:32:10,836 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=10, state=RUNNABLE; OpenRegionProcedure 2cba8ed741e888a6ec6550e0722fbc39, server=jenkins-hbase20.apache.org,33475,1689121928349}] 2023-07-12 00:32:10,836 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=11, state=RUNNABLE; OpenRegionProcedure 977900e03dd851ff67f68c586e17ff0e, server=jenkins-hbase20.apache.org,37887,1689121929610}] 2023-07-12 00:32:10,840 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=14, state=RUNNABLE; OpenRegionProcedure e05a790448a127b53382088bce1a74d6, server=jenkins-hbase20.apache.org,33475,1689121928349}] 2023-07-12 00:32:10,985 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,37887,1689121929610 2023-07-12 00:32:10,985 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 00:32:10,986 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1689121929714.38818f72237f4c84213ef9820649179c. 2023-07-12 00:32:10,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 38818f72237f4c84213ef9820649179c, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1689121929714.38818f72237f4c84213ef9820649179c.', STARTKEY => '5', ENDKEY => ''} 2023-07-12 00:32:10,987 INFO [RS-EventLoopGroup-19-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57494, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 00:32:10,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testLogSplittingAfterMasterRecoveryDueToZKExpiry 38818f72237f4c84213ef9820649179c 2023-07-12 00:32:10,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1689121929714.38818f72237f4c84213ef9820649179c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:10,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 38818f72237f4c84213ef9820649179c 2023-07-12 00:32:10,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 38818f72237f4c84213ef9820649179c 2023-07-12 00:32:10,989 INFO [StoreOpener-38818f72237f4c84213ef9820649179c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family col of region 38818f72237f4c84213ef9820649179c 2023-07-12 00:32:10,991 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1689121929714.02432dc8a0cc182e04b7721d9cac70d0. 2023-07-12 00:32:10,991 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 02432dc8a0cc182e04b7721d9cac70d0, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1689121929714.02432dc8a0cc182e04b7721d9cac70d0.', STARTKEY => '3', ENDKEY => '4'} 2023-07-12 00:32:10,991 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testLogSplittingAfterMasterRecoveryDueToZKExpiry 02432dc8a0cc182e04b7721d9cac70d0 2023-07-12 00:32:10,991 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1689121929714.02432dc8a0cc182e04b7721d9cac70d0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:10,991 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 02432dc8a0cc182e04b7721d9cac70d0 2023-07-12 00:32:10,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 02432dc8a0cc182e04b7721d9cac70d0 2023-07-12 00:32:10,992 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39. 2023-07-12 00:32:10,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2cba8ed741e888a6ec6550e0722fbc39, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39.', STARTKEY => '', ENDKEY => '1'} 2023-07-12 00:32:10,992 DEBUG [StoreOpener-38818f72237f4c84213ef9820649179c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/38818f72237f4c84213ef9820649179c/col 2023-07-12 00:32:10,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testLogSplittingAfterMasterRecoveryDueToZKExpiry 2cba8ed741e888a6ec6550e0722fbc39 2023-07-12 00:32:10,992 DEBUG [StoreOpener-38818f72237f4c84213ef9820649179c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/38818f72237f4c84213ef9820649179c/col 2023-07-12 00:32:10,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:10,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 2cba8ed741e888a6ec6550e0722fbc39 2023-07-12 00:32:10,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 2cba8ed741e888a6ec6550e0722fbc39 2023-07-12 00:32:10,993 INFO [StoreOpener-02432dc8a0cc182e04b7721d9cac70d0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family col of region 02432dc8a0cc182e04b7721d9cac70d0 2023-07-12 00:32:10,993 INFO [StoreOpener-38818f72237f4c84213ef9820649179c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 38818f72237f4c84213ef9820649179c columnFamilyName col 2023-07-12 00:32:10,994 INFO [StoreOpener-38818f72237f4c84213ef9820649179c-1] regionserver.HStore(310): Store=38818f72237f4c84213ef9820649179c/col, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:10,994 INFO [StoreOpener-2cba8ed741e888a6ec6550e0722fbc39-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family col of region 2cba8ed741e888a6ec6550e0722fbc39 2023-07-12 00:32:10,995 DEBUG [StoreOpener-02432dc8a0cc182e04b7721d9cac70d0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/02432dc8a0cc182e04b7721d9cac70d0/col 2023-07-12 00:32:10,995 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/38818f72237f4c84213ef9820649179c 2023-07-12 00:32:10,995 DEBUG [StoreOpener-02432dc8a0cc182e04b7721d9cac70d0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/02432dc8a0cc182e04b7721d9cac70d0/col 2023-07-12 00:32:10,995 DEBUG [StoreOpener-2cba8ed741e888a6ec6550e0722fbc39-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39/col 2023-07-12 00:32:10,995 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/38818f72237f4c84213ef9820649179c 2023-07-12 00:32:10,995 DEBUG [StoreOpener-2cba8ed741e888a6ec6550e0722fbc39-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39/col 2023-07-12 00:32:10,995 INFO [StoreOpener-02432dc8a0cc182e04b7721d9cac70d0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 02432dc8a0cc182e04b7721d9cac70d0 columnFamilyName col 2023-07-12 00:32:10,996 INFO [StoreOpener-2cba8ed741e888a6ec6550e0722fbc39-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2cba8ed741e888a6ec6550e0722fbc39 columnFamilyName col 2023-07-12 00:32:10,996 INFO [StoreOpener-02432dc8a0cc182e04b7721d9cac70d0-1] regionserver.HStore(310): Store=02432dc8a0cc182e04b7721d9cac70d0/col, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:10,997 INFO [StoreOpener-2cba8ed741e888a6ec6550e0722fbc39-1] regionserver.HStore(310): Store=2cba8ed741e888a6ec6550e0722fbc39/col, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:10,997 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/02432dc8a0cc182e04b7721d9cac70d0 2023-07-12 00:32:10,997 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/02432dc8a0cc182e04b7721d9cac70d0 2023-07-12 00:32:10,998 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39 2023-07-12 00:32:10,998 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39 2023-07-12 00:32:10,998 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 38818f72237f4c84213ef9820649179c 2023-07-12 00:32:11,000 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 02432dc8a0cc182e04b7721d9cac70d0 2023-07-12 00:32:11,002 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 2cba8ed741e888a6ec6550e0722fbc39 2023-07-12 00:32:11,004 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/38818f72237f4c84213ef9820649179c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:11,004 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/02432dc8a0cc182e04b7721d9cac70d0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:11,004 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 38818f72237f4c84213ef9820649179c; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64379203, jitterRate=-0.04067511856555939}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:11,005 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 38818f72237f4c84213ef9820649179c: 2023-07-12 00:32:11,005 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 02432dc8a0cc182e04b7721d9cac70d0; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64426828, jitterRate=-0.03996545076370239}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:11,005 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 02432dc8a0cc182e04b7721d9cac70d0: 2023-07-12 00:32:11,005 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:11,005 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1689121929714.38818f72237f4c84213ef9820649179c., pid=16, masterSystemTime=1689121930982 2023-07-12 00:32:11,006 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1689121929714.02432dc8a0cc182e04b7721d9cac70d0., pid=18, masterSystemTime=1689121930985 2023-07-12 00:32:11,006 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 2cba8ed741e888a6ec6550e0722fbc39; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61230295, jitterRate=-0.08759750425815582}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:11,006 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 2cba8ed741e888a6ec6550e0722fbc39: 2023-07-12 00:32:11,009 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39., pid=19, masterSystemTime=1689121930988 2023-07-12 00:32:11,009 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1689121929714.38818f72237f4c84213ef9820649179c. 2023-07-12 00:32:11,009 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1689121929714.38818f72237f4c84213ef9820649179c. 2023-07-12 00:32:11,009 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1689121929714.810f496a1e86b8d9ebefb6030254ac4d. 2023-07-12 00:32:11,009 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 810f496a1e86b8d9ebefb6030254ac4d, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1689121929714.810f496a1e86b8d9ebefb6030254ac4d.', STARTKEY => '2', ENDKEY => '3'} 2023-07-12 00:32:11,010 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testLogSplittingAfterMasterRecoveryDueToZKExpiry 810f496a1e86b8d9ebefb6030254ac4d 2023-07-12 00:32:11,010 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=38818f72237f4c84213ef9820649179c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:11,010 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1689121929714.810f496a1e86b8d9ebefb6030254ac4d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:11,010 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1689121929714.02432dc8a0cc182e04b7721d9cac70d0. 2023-07-12 00:32:11,010 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 810f496a1e86b8d9ebefb6030254ac4d 2023-07-12 00:32:11,011 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1689121929714.02432dc8a0cc182e04b7721d9cac70d0. 2023-07-12 00:32:11,010 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1689121929714.38818f72237f4c84213ef9820649179c.","families":{"info":[{"qualifier":"regioninfo","vlen":83,"tag":[],"timestamp":"1689121931010"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121931010"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121931010"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121931010"}]},"ts":"1689121931010"} 2023-07-12 00:32:11,011 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1689121929714.977900e03dd851ff67f68c586e17ff0e. 2023-07-12 00:32:11,011 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 810f496a1e86b8d9ebefb6030254ac4d 2023-07-12 00:32:11,011 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 977900e03dd851ff67f68c586e17ff0e, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1689121929714.977900e03dd851ff67f68c586e17ff0e.', STARTKEY => '1', ENDKEY => '2'} 2023-07-12 00:32:11,011 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=02432dc8a0cc182e04b7721d9cac70d0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,37887,1689121929610 2023-07-12 00:32:11,011 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1689121929714.02432dc8a0cc182e04b7721d9cac70d0.","families":{"info":[{"qualifier":"regioninfo","vlen":84,"tag":[],"timestamp":"1689121931011"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121931011"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121931011"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121931011"}]},"ts":"1689121931011"} 2023-07-12 00:32:11,011 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testLogSplittingAfterMasterRecoveryDueToZKExpiry 977900e03dd851ff67f68c586e17ff0e 2023-07-12 00:32:11,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1689121929714.977900e03dd851ff67f68c586e17ff0e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:11,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39. 2023-07-12 00:32:11,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 977900e03dd851ff67f68c586e17ff0e 2023-07-12 00:32:11,012 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39. 2023-07-12 00:32:11,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 977900e03dd851ff67f68c586e17ff0e 2023-07-12 00:32:11,012 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6. 2023-07-12 00:32:11,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e05a790448a127b53382088bce1a74d6, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6.', STARTKEY => '4', ENDKEY => '5'} 2023-07-12 00:32:11,012 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=2cba8ed741e888a6ec6550e0722fbc39, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,33475,1689121928349 2023-07-12 00:32:11,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testLogSplittingAfterMasterRecoveryDueToZKExpiry e05a790448a127b53382088bce1a74d6 2023-07-12 00:32:11,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:11,012 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39.","families":{"info":[{"qualifier":"regioninfo","vlen":83,"tag":[],"timestamp":"1689121931012"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121931012"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121931012"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121931012"}]},"ts":"1689121931012"} 2023-07-12 00:32:11,013 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for e05a790448a127b53382088bce1a74d6 2023-07-12 00:32:11,013 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for e05a790448a127b53382088bce1a74d6 2023-07-12 00:32:11,013 INFO [StoreOpener-977900e03dd851ff67f68c586e17ff0e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family col of region 977900e03dd851ff67f68c586e17ff0e 2023-07-12 00:32:11,013 INFO [StoreOpener-810f496a1e86b8d9ebefb6030254ac4d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family col of region 810f496a1e86b8d9ebefb6030254ac4d 2023-07-12 00:32:11,014 INFO [StoreOpener-e05a790448a127b53382088bce1a74d6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family col of region e05a790448a127b53382088bce1a74d6 2023-07-12 00:32:11,016 DEBUG [StoreOpener-810f496a1e86b8d9ebefb6030254ac4d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/810f496a1e86b8d9ebefb6030254ac4d/col 2023-07-12 00:32:11,016 DEBUG [StoreOpener-810f496a1e86b8d9ebefb6030254ac4d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/810f496a1e86b8d9ebefb6030254ac4d/col 2023-07-12 00:32:11,017 INFO [StoreOpener-810f496a1e86b8d9ebefb6030254ac4d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 810f496a1e86b8d9ebefb6030254ac4d columnFamilyName col 2023-07-12 00:32:11,017 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-07-12 00:32:11,018 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; OpenRegionProcedure 38818f72237f4c84213ef9820649179c, server=jenkins-hbase20.apache.org,42069,1689121928319 in 185 msec 2023-07-12 00:32:11,018 INFO [StoreOpener-810f496a1e86b8d9ebefb6030254ac4d-1] regionserver.HStore(310): Store=810f496a1e86b8d9ebefb6030254ac4d/col, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:11,018 DEBUG [StoreOpener-e05a790448a127b53382088bce1a74d6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/e05a790448a127b53382088bce1a74d6/col 2023-07-12 00:32:11,018 DEBUG [StoreOpener-e05a790448a127b53382088bce1a74d6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/e05a790448a127b53382088bce1a74d6/col 2023-07-12 00:32:11,018 DEBUG [StoreOpener-977900e03dd851ff67f68c586e17ff0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/977900e03dd851ff67f68c586e17ff0e/col 2023-07-12 00:32:11,018 DEBUG [StoreOpener-977900e03dd851ff67f68c586e17ff0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/977900e03dd851ff67f68c586e17ff0e/col 2023-07-12 00:32:11,018 INFO [StoreOpener-e05a790448a127b53382088bce1a74d6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e05a790448a127b53382088bce1a74d6 columnFamilyName col 2023-07-12 00:32:11,019 INFO [StoreOpener-977900e03dd851ff67f68c586e17ff0e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 977900e03dd851ff67f68c586e17ff0e columnFamilyName col 2023-07-12 00:32:11,019 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/810f496a1e86b8d9ebefb6030254ac4d 2023-07-12 00:32:11,019 INFO [StoreOpener-e05a790448a127b53382088bce1a74d6-1] regionserver.HStore(310): Store=e05a790448a127b53382088bce1a74d6/col, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:11,019 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=13 2023-07-12 00:32:11,019 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=38818f72237f4c84213ef9820649179c, ASSIGN in 350 msec 2023-07-12 00:32:11,019 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=13, state=SUCCESS; OpenRegionProcedure 02432dc8a0cc182e04b7721d9cac70d0, server=jenkins-hbase20.apache.org,37887,1689121929610 in 183 msec 2023-07-12 00:32:11,020 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/810f496a1e86b8d9ebefb6030254ac4d 2023-07-12 00:32:11,020 INFO [StoreOpener-977900e03dd851ff67f68c586e17ff0e-1] regionserver.HStore(310): Store=977900e03dd851ff67f68c586e17ff0e/col, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:11,020 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=10 2023-07-12 00:32:11,020 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=10, state=SUCCESS; OpenRegionProcedure 2cba8ed741e888a6ec6550e0722fbc39, server=jenkins-hbase20.apache.org,33475,1689121928349 in 181 msec 2023-07-12 00:32:11,020 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/e05a790448a127b53382088bce1a74d6 2023-07-12 00:32:11,021 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/e05a790448a127b53382088bce1a74d6 2023-07-12 00:32:11,021 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/977900e03dd851ff67f68c586e17ff0e 2023-07-12 00:32:11,021 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=02432dc8a0cc182e04b7721d9cac70d0, ASSIGN in 351 msec 2023-07-12 00:32:11,021 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/977900e03dd851ff67f68c586e17ff0e 2023-07-12 00:32:11,022 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=2cba8ed741e888a6ec6550e0722fbc39, ASSIGN in 352 msec 2023-07-12 00:32:11,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 810f496a1e86b8d9ebefb6030254ac4d 2023-07-12 00:32:11,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for e05a790448a127b53382088bce1a74d6 2023-07-12 00:32:11,024 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 977900e03dd851ff67f68c586e17ff0e 2023-07-12 00:32:11,026 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/810f496a1e86b8d9ebefb6030254ac4d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:11,027 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/e05a790448a127b53382088bce1a74d6/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:11,027 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 810f496a1e86b8d9ebefb6030254ac4d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69579695, jitterRate=0.0368182510137558}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:11,027 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 810f496a1e86b8d9ebefb6030254ac4d: 2023-07-12 00:32:11,027 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/977900e03dd851ff67f68c586e17ff0e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:11,027 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened e05a790448a127b53382088bce1a74d6; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65281159, jitterRate=-0.02723492681980133}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:11,027 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for e05a790448a127b53382088bce1a74d6: 2023-07-12 00:32:11,028 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 977900e03dd851ff67f68c586e17ff0e; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72126513, jitterRate=0.0747687965631485}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:11,028 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 977900e03dd851ff67f68c586e17ff0e: 2023-07-12 00:32:11,028 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1689121929714.977900e03dd851ff67f68c586e17ff0e., pid=20, masterSystemTime=1689121930985 2023-07-12 00:32:11,032 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1689121929714.810f496a1e86b8d9ebefb6030254ac4d., pid=17, masterSystemTime=1689121930982 2023-07-12 00:32:11,032 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6., pid=21, masterSystemTime=1689121930988 2023-07-12 00:32:11,032 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1689121929714.977900e03dd851ff67f68c586e17ff0e. 2023-07-12 00:32:11,032 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1689121929714.977900e03dd851ff67f68c586e17ff0e. 2023-07-12 00:32:11,033 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=977900e03dd851ff67f68c586e17ff0e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,37887,1689121929610 2023-07-12 00:32:11,033 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1689121929714.977900e03dd851ff67f68c586e17ff0e.","families":{"info":[{"qualifier":"regioninfo","vlen":84,"tag":[],"timestamp":"1689121931033"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121931033"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121931033"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121931033"}]},"ts":"1689121931033"} 2023-07-12 00:32:11,033 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6. 2023-07-12 00:32:11,034 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6. 2023-07-12 00:32:11,034 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=e05a790448a127b53382088bce1a74d6, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,33475,1689121928349 2023-07-12 00:32:11,034 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6.","families":{"info":[{"qualifier":"regioninfo","vlen":84,"tag":[],"timestamp":"1689121931034"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121931034"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121931034"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121931034"}]},"ts":"1689121931034"} 2023-07-12 00:32:11,035 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=810f496a1e86b8d9ebefb6030254ac4d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:11,035 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1689121929714.810f496a1e86b8d9ebefb6030254ac4d.","families":{"info":[{"qualifier":"regioninfo","vlen":84,"tag":[],"timestamp":"1689121931035"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121931035"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121931035"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121931035"}]},"ts":"1689121931035"} 2023-07-12 00:32:11,035 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1689121929714.810f496a1e86b8d9ebefb6030254ac4d. 2023-07-12 00:32:11,035 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1689121929714.810f496a1e86b8d9ebefb6030254ac4d. 2023-07-12 00:32:11,037 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=11 2023-07-12 00:32:11,038 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=11, state=SUCCESS; OpenRegionProcedure 977900e03dd851ff67f68c586e17ff0e, server=jenkins-hbase20.apache.org,37887,1689121929610 in 199 msec 2023-07-12 00:32:11,038 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=14 2023-07-12 00:32:11,038 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=14, state=SUCCESS; OpenRegionProcedure e05a790448a127b53382088bce1a74d6, server=jenkins-hbase20.apache.org,33475,1689121928349 in 196 msec 2023-07-12 00:32:11,039 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=12 2023-07-12 00:32:11,039 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=977900e03dd851ff67f68c586e17ff0e, ASSIGN in 370 msec 2023-07-12 00:32:11,039 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=12, state=SUCCESS; OpenRegionProcedure 810f496a1e86b8d9ebefb6030254ac4d, server=jenkins-hbase20.apache.org,42069,1689121928319 in 207 msec 2023-07-12 00:32:11,042 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=e05a790448a127b53382088bce1a74d6, ASSIGN in 370 msec 2023-07-12 00:32:11,043 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=12, resume processing ppid=9 2023-07-12 00:32:11,043 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=810f496a1e86b8d9ebefb6030254ac4d, ASSIGN in 371 msec 2023-07-12 00:32:11,044 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 00:32:11,044 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689121931044"}]},"ts":"1689121931044"} 2023-07-12 00:32:11,046 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testLogSplittingAfterMasterRecoveryDueToZKExpiry, state=ENABLED in hbase:meta 2023-07-12 00:32:11,048 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 00:32:11,049 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry in 1.3330 sec 2023-07-12 00:32:11,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42265] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 00:32:11,832 INFO [Listener at localhost.localdomain/34481] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testLogSplittingAfterMasterRecoveryDueToZKExpiry, procId: 9 completed 2023-07-12 00:32:11,835 INFO [Listener at localhost.localdomain/34481] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 00:32:11,850 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 00:32:11,851 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(159): ***** ABORTING master jenkins-hbase20.apache.org,42265,1689121928263: Test recovery from zk session expired ***** org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.TestZooKeeper.testLogSplittingAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:270) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:32:11,851 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,42265,1689121928263' ***** 2023-07-12 00:32:11,851 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2311): STOPPED: Stopped by Listener at localhost.localdomain/34481 2023-07-12 00:32:11,851 DEBUG [M:0;jenkins-hbase20:42265] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4e5a7c3c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-12 00:32:11,851 INFO [M:0;jenkins-hbase20:42265] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,42265,1689121928263 2023-07-12 00:32:11,852 DEBUG [M:0;jenkins-hbase20:42265] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2efaee1a to 127.0.0.1:58965 2023-07-12 00:32:11,852 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:11,852 DEBUG [M:0;jenkins-hbase20:42265] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:11,852 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:37887-0x1007e9741810026, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:11,852 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:11,852 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:11,852 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:33475-0x1007e9741810021, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:11,852 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:11,852 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:11,852 INFO [M:0;jenkins-hbase20:42265] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,42265,1689121928263; all regions closed. 2023-07-12 00:32:11,853 DEBUG [M:0;jenkins-hbase20:42265] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:11,853 DEBUG [M:0;jenkins-hbase20:42265] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 00:32:11,853 DEBUG [M:0;jenkins-hbase20:42265] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 00:32:11,853 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 00:32:11,853 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689121928536] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689121928536,5,FailOnTimeoutGroup] 2023-07-12 00:32:11,855 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689121928540] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689121928540,5,FailOnTimeoutGroup] 2023-07-12 00:32:11,856 INFO [M:0;jenkins-hbase20:42265] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 00:32:11,856 INFO [M:0;jenkins-hbase20:42265] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 00:32:11,856 INFO [M:0;jenkins-hbase20:42265] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-07-12 00:32:11,856 DEBUG [M:0;jenkins-hbase20:42265] master.HMaster(1512): Stopping service threads 2023-07-12 00:32:11,856 INFO [M:0;jenkins-hbase20:42265] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 00:32:11,857 INFO [M:0;jenkins-hbase20:42265] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 00:32:11,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:11,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:11,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37887-0x1007e9741810026, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:11,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33475-0x1007e9741810021, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:11,858 ERROR [M:0;jenkins-hbase20:42265] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-7,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-07-12 00:32:11,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:11,858 INFO [M:0;jenkins-hbase20:42265] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=true 2023-07-12 00:32:11,859 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 00:32:11,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:11,859 DEBUG [zk-event-processor-pool-0] master.ActiveMasterManager(186): No master available. Notifying waiting threads 2023-07-12 00:32:11,859 DEBUG [M:0;jenkins-hbase20:42265] zookeeper.ZKUtil(398): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 00:32:11,859 WARN [M:0;jenkins-hbase20:42265] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 00:32:11,859 INFO [M:0;jenkins-hbase20:42265] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 00:32:11,860 INFO [M:0;jenkins-hbase20:42265] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=true 2023-07-12 00:32:11,861 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:11,861 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:37887-0x1007e9741810026, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:11,861 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:11,861 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:33475-0x1007e9741810021, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:11,861 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:11,861 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:11,861 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:11,861 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:32:11,864 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,45743,1689121928288 from backup master directory 2023-07-12 00:32:11,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:32:11,864 DEBUG [Listener at localhost.localdomain/34481] hbase.HBaseTestingUtility(2627): Found 6 regions for table testLogSplittingAfterMasterRecoveryDueToZKExpiry 2023-07-12 00:32:11,864 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:32:11,865 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 00:32:11,865 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,45743,1689121928288 2023-07-12 00:32:11,865 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,45743,1689121928288 2023-07-12 00:32:11,865 DEBUG [Listener at localhost.localdomain/34481] hbase.HBaseTestingUtility(2633): firstRegionName=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39. 2023-07-12 00:32:11,865 INFO [Listener at localhost.localdomain/34481] hbase.MiniHBaseCluster(271): Killing jenkins-hbase20.apache.org,33475,1689121928349 2023-07-12 00:32:11,865 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:32:11,866 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase20.apache.org,33475,1689121928349: Simulated kill ***** 2023-07-12 00:32:11,866 DEBUG [M:0;jenkins-hbase20:42265] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 00:32:11,866 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 00:32:11,867 INFO [M:0;jenkins-hbase20:42265] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:11,867 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-12 00:32:11,867 DEBUG [M:0;jenkins-hbase20:42265] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:11,867 DEBUG [M:0;jenkins-hbase20:42265] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 00:32:11,867 DEBUG [M:0;jenkins-hbase20:42265] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:11,867 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-12 00:32:11,868 ERROR [M:0;jenkins-hbase20:42265] regionserver.HRegion(1825): Memstore data size is 75813 in region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:11,868 INFO [M:0;jenkins-hbase20:42265] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:11,868 DEBUG [M:0;jenkins-hbase20:42265] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 00:32:11,868 INFO [M:0;jenkins-hbase20:42265] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 00:32:11,868 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 00:32:11,868 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-12 00:32:11,869 INFO [M:0;jenkins-hbase20:42265] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:42265 2023-07-12 00:32:11,870 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-12 00:32:11,872 DEBUG [M:0;jenkins-hbase20:42265] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,42265,1689121928263 already deleted, retry=false 2023-07-12 00:32:11,874 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1192755200, "init": 524288000, "max": 2051014656, "used": 326746520 }, "NonHeapMemoryUsage": { "committed": 138567680, "init": 2555904, "max": -1, "used": 136129616 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 0, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 27532, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 0, "ProcessCallTime_num_ops": 114, "ProcessCallTime_min": 0, "ProcessCallTime_max": 6, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 0, "ProcessCallTime_median": 1, "ProcessCallTime_75th_percentile": 1, "ProcessCallTime_90th_percentile": 3, "ProcessCallTime_95th_percentile": 4, "ProcessCallTime_98th_percentile": 6, "ProcessCallTime_99th_percentile": 6, "ProcessCallTime_99.9th_percentile": 6, "ProcessCallTime_TimeRangeCount_0-1": 46, "ProcessCallTime_TimeRangeCount_1-3": 8, "ProcessCallTime_TimeRangeCount_3-10": 5, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 114, "QueueCallTime_min": 0, "QueueCallTime_max": 1, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 0, "QueueCallTime_99th_percentile": 1, "QueueCallTime_99.9th_percentile": 1, "QueueCallTime_TimeRangeCount_0-1": 59, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 114, "TotalCallTime_min": 0, "TotalCallTime_max": 6, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 0, "TotalCallTime_median": 1, "TotalCallTime_75th_percentile": 1, "TotalCallTime_90th_percentile": 3, "TotalCallTime_95th_percentile": 4, "TotalCallTime_98th_percentile": 6, "TotalCallTime_99th_percentile": 6, "TotalCallTime_99.9th_percentile": 6, "TotalCallTime_TimeRangeCount_0-1": 45, "TotalCallTime_TimeRangeCount_1-3": 9, "TotalCallTime_TimeRangeCount_3-10": 5, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 114, "ResponseSize_min": 0, "ResponseSize_max": 1011, "ResponseSize_mean": 155, "ResponseSize_25th_percentile": 1, "ResponseSize_median": 5, "ResponseSize_75th_percentile": 74, "ResponseSize_90th_percentile": 771, "ResponseSize_95th_percentile": 824, "ResponseSize_98th_percentile": 968, "ResponseSize_99th_percentile": 989, "ResponseSize_99.9th_percentile": 1008, "ResponseSize_SizeRangeCount_0-10": 32, "ResponseSize_SizeRangeCount_10-100": 14, "ResponseSize_SizeRangeCount_100-1000": 13, "exceptions.UnknownScannerException": 0, "exceptions": 0, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 0, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 114, "RequestSize_min": 80, "RequestSize_max": 414, "RequestSize_mean": 152, "RequestSize_25th_percentile": 96, "RequestSize_median": 121, "RequestSize_75th_percentile": 154, "RequestSize_90th_percentile": 291, "RequestSize_95th_percentile": 343, "RequestSize_98th_percentile": 385, "RequestSize_99th_percentile": 399, "RequestSize_99.9th_percentile": 412, "RequestSize_SizeRangeCount_10-100": 20, "RequestSize_SizeRangeCount_100-1000": 39, "sentBytes": 14869 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:58965", "tag.serverName": "jenkins-hbase20.apache.org,42069,1689121928319", "tag.clusterId": "b855a1de-f551-45f5-8b9b-f7b8ff57b75f", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "regionCount": 0, "storeCount": 0, "hlogFileCount": 0, "hlogFileSize": 0, "storeFileCount": 0, "maxStoreFileCount": 0, "memStoreSize": 0, "storeFileSize": 0, "maxStoreFileAge": 0, "minStoreFileAge": 0, "avgStoreFileAge": 0, "numReferenceFiles": 0, "regionServerStartTime": 1689121928319, "averageRegionSize": 0, "storeFileIndexSize": 0, "staticIndexSize": 0, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 72812, "localBytesRead": 72812, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803968, "blockCacheCount": 0, "blockCacheDataBlockCount": 0, "blockCacheSize": 601920, "blockCacheCountHitPercent": 0.0, "blockCacheExpressHitPercent": 0.0, "l1CacheSize": 601920, "l1CacheFreeSize": 819803968, "l1CacheCount": 0, "l1CacheEvictionCount": 0, "l1CacheHitCount": 0, "l1CacheMissCount": 0, "l1CacheHitRatio": 0.0, "l1CacheMissRatio": 0.0, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 57682, "ByteBuffAllocatorPoolAllocationBytes": 0, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 0, "totalRequestCount": 0, "totalRowActionRequestCount": 0, "readRequestCount": 0, "filteredReadRequestCount": 0, "writeRequestCount": 0, "rpcGetRequestCount": 0, "rpcFullScanRequestCount": 0, "rpcScanRequestCount": 0, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 0, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 0, "blockCacheHitCountPrimary": 0, "blockCacheHitCachingCount": 0, "blockCacheMissCount": 0, "blockCacheMissCountPrimary": 0, "blockCacheMissCachingCount": 0, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 0, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 0, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 0, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 0, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 0, "ScanTime_min": 0, "ScanTime_max": 0, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 0, "ScanTime_75th_percentile": 0, "ScanTime_90th_percentile": 0, "ScanTime_95th_percentile": 0, "ScanTime_98th_percentile": 0, "ScanTime_99th_percentile": 0, "ScanTime_99.9th_percentile": 0, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 0, "Put_min": 0, "Put_max": 0, "Put_mean": 0, "Put_25th_percentile": 0, "Put_median": 0, "Put_75th_percentile": 0, "Put_90th_percentile": 0, "Put_95th_percentile": 0, "Put_98th_percentile": 0, "Put_99th_percentile": 0, "Put_99.9th_percentile": 0, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 0, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 0, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 0, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 0, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 0, "ScanSize_min": 0, "ScanSize_max": 0, "ScanSize_mean": 0, "ScanSize_25th_percentile": 0, "ScanSize_median": 0, "ScanSize_75th_percentile": 0, "ScanSize_90th_percentile": 0, "ScanSize_95th_percentile": 0, "ScanSize_98th_percentile": 0, "ScanSize_99th_percentile": 0, "ScanSize_99.9th_percentile": 0, "slowGetCount": 0, "flushedMemstoreBytes": 0, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 0, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 67, "ServerReadQueryPerSecond_mean_rate": 2.648749174359828, "ServerReadQueryPerSecond_1min_rate": 2.6105666662895697, "ServerReadQueryPerSecond_5min_rate": 2.5953241407270244, "ServerReadQueryPerSecond_15min_rate": 2.598003742790458, "ServerWriteQueryPerSecond_count": 41, "ServerWriteQueryPerSecond_mean_rate": 1.6208763325547715, "ServerWriteQueryPerSecond_1min_rate": 1.494770352860912, "ServerWriteQueryPerSecond_5min_rate": 1.4197805199423947, "ServerWriteQueryPerSecond_15min_rate": 1.4066420777463897 } ] } 2023-07-12 00:32:11,875 WARN [RS-EventLoopGroup-18-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase20.apache.org/148.251.75.209:42265 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:42265 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:32:11,875 WARN [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2576): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.net.ConnectException: Call to address=jenkins-hbase20.apache.org/148.251.75.209:42265 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:42265 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:334) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:581) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:16481) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2573) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:196) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$200(MiniHBaseCluster.java:123) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:189) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:186) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2597) at org.apache.hadoop.hbase.regionserver.HRegionServer.kill(HRegionServer.java:2612) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.kill(MiniHBaseCluster.java:181) at org.apache.hadoop.hbase.MiniHBaseCluster.killRegionServer(MiniHBaseCluster.java:272) at org.apache.hadoop.hbase.TestZooKeeper.testLogSplittingAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:273) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) Caused by: java.net.ConnectException: Call to address=jenkins-hbase20.apache.org/148.251.75.209:42265 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:42265 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:186) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:385) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:99) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:398) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:368) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1428) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:396) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:913) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:195) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$300(NettyRpcConnection.java:76) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:296) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:287) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:674) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:693) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:42265 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:32:11,875 DEBUG [RS-EventLoopGroup-18-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase20.apache.org/148.251.75.209:42265 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:42265 2023-07-12 00:32:11,877 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,33475,1689121928349' ***** 2023-07-12 00:32:11,877 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2311): STOPPED: Simulated kill 2023-07-12 00:32:11,878 INFO [RS:1;jenkins-hbase20:33475] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 00:32:11,878 INFO [RS:1;jenkins-hbase20:33475] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-12 00:32:11,878 INFO [RS:1;jenkins-hbase20:33475] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-12 00:32:11,878 DEBUG [RS:1;jenkins-hbase20:33475] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x01a1e03e to 127.0.0.1:58965 2023-07-12 00:32:11,878 DEBUG [RS:1;jenkins-hbase20:33475] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:11,878 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 00:32:11,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33475] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,33475,1689121928349 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:11,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33475] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Scan size: 171 connection: 148.251.75.209:49206 deadline: 1689121991878, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,33475,1689121928349 aborting 2023-07-12 00:32:11,889 DEBUG [RS:1;jenkins-hbase20:33475] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:11,890 INFO [RS:1;jenkins-hbase20:33475] regionserver.LeaseManager(133): Closed leases 2023-07-12 00:32:11,891 INFO [RS:1;jenkins-hbase20:33475] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:33475 2023-07-12 00:32:11,893 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:11,894 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:33475-0x1007e9741810021, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,33475,1689121928349 2023-07-12 00:32:11,894 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:37887-0x1007e9741810026, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,33475,1689121928349 2023-07-12 00:32:11,894 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:33475-0x1007e9741810021, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:11,894 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,33475,1689121928349 2023-07-12 00:32:11,894 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:11,894 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:37887-0x1007e9741810026, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:11,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,37887,1689121929610 2023-07-12 00:32:11,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37887-0x1007e9741810026, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,37887,1689121929610 2023-07-12 00:32:11,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:11,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37887-0x1007e9741810026, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:11,896 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase20.apache.org,33475,1689121928349 znode expired, triggering replicatorRemoved event 2023-07-12 00:32:11,896 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase20.apache.org,33475,1689121928349 znode expired, triggering replicatorRemoved event 2023-07-12 00:32:11,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37887-0x1007e9741810026, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,37887,1689121929610 2023-07-12 00:32:11,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,37887,1689121929610 2023-07-12 00:32:11,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:11,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37887-0x1007e9741810026, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:11,923 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x18bc20ee to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:32:11,926 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2fbbbad6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:32:11,926 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 00:32:11,926 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 00:32:11,927 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:32:11,934 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,42265,1689121928263 to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,42265,1689121928263-dead as it is dead 2023-07-12 00:32:11,934 INFO [master/jenkins-hbase20:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,42265,1689121928263-dead/jenkins-hbase20.apache.org%2C42265%2C1689121928263.1689121928453 2023-07-12 00:32:11,935 INFO [master/jenkins-hbase20:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,42265,1689121928263-dead/jenkins-hbase20.apache.org%2C42265%2C1689121928263.1689121928453 after 1ms 2023-07-12 00:32:11,935 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,42265,1689121928263-dead/jenkins-hbase20.apache.org%2C42265%2C1689121928263.1689121928453 to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase20.apache.org%2C42265%2C1689121928263.1689121928453 2023-07-12 00:32:11,935 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,42265,1689121928263-dead 2023-07-12 00:32:11,936 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,45743,1689121928288 2023-07-12 00:32:11,938 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C45743%2C1689121928288, suffix=, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,45743,1689121928288, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/oldWALs, maxLogs=10 2023-07-12 00:32:11,945 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 00:32:11,952 DEBUG [RS-EventLoopGroup-19-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:32:11,952 DEBUG [RS-EventLoopGroup-19-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:32:11,954 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,45743,1689121928288/jenkins-hbase20.apache.org%2C45743%2C1689121928288.1689121931938 2023-07-12 00:32:11,954 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK], DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK]] 2023-07-12 00:32:11,954 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 00:32:11,954 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:11,955 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:11,955 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:11,956 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:11,958 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 00:32:11,960 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 00:32:11,961 WARN [RS-EventLoopGroup-18-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase20.apache.org/148.251.75.209:42265 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:42265 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:32:11,961 DEBUG [RS-EventLoopGroup-18-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase20.apache.org/148.251.75.209:42265 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:42265 2023-07-12 00:32:11,962 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:11,962 WARN [RS-EventLoopGroup-19-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase20.apache.org/148.251.75.209:42265 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:42265 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:32:11,962 DEBUG [RS-EventLoopGroup-19-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase20.apache.org/148.251.75.209:42265 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:42265 2023-07-12 00:32:11,963 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5179): Found 1 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-07-12 00:32:11,963 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5276): Replaying edits from hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase20.apache.org%2C42265%2C1689121928263.1689121928453 2023-07-12 00:32:11,975 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5464): Applied 185, skipped 0, firstSequenceIdInLog=3, maxSequenceIdInLog=155, path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase20.apache.org%2C42265%2C1689121928263.1689121928453 2023-07-12 00:32:11,976 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=74.04 KB heapSize=87.93 KB 2023-07-12 00:32:11,985 WARN [RPCClient-NioEventLoopGroup-6-10] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase20.apache.org/148.251.75.209:33475 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:33475 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:32:11,985 DEBUG [RPCClient-NioEventLoopGroup-6-10] ipc.FailedServers(52): Added failed server with address jenkins-hbase20.apache.org/148.251.75.209:33475 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:33475 2023-07-12 00:32:12,004 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=74.04 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/28c67f08bccd4e65a59a03ed6a6fd539 2023-07-12 00:32:12,019 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/28c67f08bccd4e65a59a03ed6a6fd539 as hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/28c67f08bccd4e65a59a03ed6a6fd539 2023-07-12 00:32:12,029 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/28c67f08bccd4e65a59a03ed6a6fd539, entries=185, sequenceid=155, filesize=79.1 K 2023-07-12 00:32:12,030 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(2948): Finished flush of dataSize ~74.04 KB/75813, heapSize ~87.91 KB/90024, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 55ms, sequenceid=155, compaction requested=false; wal=null 2023-07-12 00:32:12,031 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase20.apache.org%2C42265%2C1689121928263.1689121928453 2023-07-12 00:32:12,038 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:12,059 INFO [RS:1;jenkins-hbase20:33475] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,33475,1689121928349; zookeeper connection closed. 2023-07-12 00:32:12,059 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:33475-0x1007e9741810021, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:12,060 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@3e702d16] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@3e702d16 2023-07-12 00:32:12,060 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:33475-0x1007e9741810021, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:12,067 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/155.seqid, newMaxSeqId=155, maxSeqId=1 2023-07-12 00:32:12,068 DEBUG [RS:2;jenkins-hbase20:37887] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 00:32:12,069 DEBUG [RS:0;jenkins-hbase20:42069] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 00:32:12,071 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=156; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66933713, jitterRate=-0.002609953284263611}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:12,071 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 00:32:12,071 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 00:32:12,073 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-12 00:32:12,074 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-12 00:32:12,074 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 00:32:12,075 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 00:32:12,075 INFO [RS-EventLoopGroup-16-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60759, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.6 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 00:32:12,076 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=45743] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:12,077 INFO [RS-EventLoopGroup-16-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37905, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.8 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 00:32:12,079 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=45743] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:12,088 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 00:32:12,090 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 00:32:12,090 DEBUG [hconnection-0x3fc558ca-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase20.apache.org/148.251.75.209:33475 this server is in the failed servers list 2023-07-12 00:32:12,090 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 00:32:12,090 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 00:32:12,092 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry 2023-07-12 00:32:12,092 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 17 msec 2023-07-12 00:32:12,092 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 00:32:12,097 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 00:32:12,098 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase20.apache.org,33475,1689121928349, table=hbase:meta, region=1588230740 2023-07-12 00:32:12,105 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 3 possibly 'live' servers, and 0 'splitting'. 2023-07-12 00:32:12,108 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,37887,1689121929610 2023-07-12 00:32:12,109 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:12,112 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,33475,1689121928349 already deleted, retry=false 2023-07-12 00:32:12,113 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(568): Processing expiration of jenkins-hbase20.apache.org,33475,1689121928349 on jenkins-hbase20.apache.org,45743,1689121928288 2023-07-12 00:32:12,119 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=22, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase20.apache.org,33475,1689121928349, splitWal=true, meta=true 2023-07-12 00:32:12,119 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=22 for jenkins-hbase20.apache.org,33475,1689121928349 (carryingMeta=true) jenkins-hbase20.apache.org,33475,1689121928349/CRASHED/regionCount=1/lock=java.util.concurrent.locks.ReentrantReadWriteLock@753f823[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-12 00:32:12,139 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 00:32:12,140 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 00:32:12,140 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 00:32:12,141 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 00:32:12,142 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 00:32:12,143 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 00:32:12,145 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 00:32:12,145 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,45743,1689121928288, sessionid=0x1007e974181001f, setting cluster-up flag (Was=true) 2023-07-12 00:32:12,147 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 00:32:12,149 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,45743,1689121928288 2023-07-12 00:32:12,155 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 00:32:12,162 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,45743,1689121928288 2023-07-12 00:32:12,163 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.hbase-snapshot/.tmp 2023-07-12 00:32:12,171 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:32:12,171 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:32:12,171 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:32:12,171 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:32:12,171 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-07-12 00:32:12,171 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:12,171 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-12 00:32:12,171 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:12,181 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=45743] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:12,202 DEBUG [PEWorker-1] master.DeadServer(103): Processing jenkins-hbase20.apache.org,33475,1689121928349; numProcessing=1 2023-07-12 00:32:12,203 INFO [PEWorker-1] procedure.ServerCrashProcedure(161): Start pid=22, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase20.apache.org,33475,1689121928349, splitWal=true, meta=true 2023-07-12 00:32:12,204 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=22, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase20.apache.org,33475,1689121928349, splitWal=true, meta=true, isMeta: true 2023-07-12 00:32:12,207 DEBUG [PEWorker-1] master.MasterWalManager(318): Renamed region directory: hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting 2023-07-12 00:32:12,209 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase20.apache.org,33475,1689121928349 WAL count=1, meta=true 2023-07-12 00:32:12,209 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase20.apache.org%2C33475%2C1689121928349.meta.1689121928922.meta}] 2023-07-12 00:32:12,219 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689121962219 2023-07-12 00:32:12,219 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 00:32:12,219 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 00:32:12,219 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 00:32:12,219 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 00:32:12,220 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 00:32:12,220 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 00:32:12,223 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:12,229 DEBUG [PEWorker-5] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase20.apache.org,37887,1689121929610 2023-07-12 00:32:12,236 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 00:32:12,236 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 00:32:12,236 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 00:32:12,237 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=45743] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 00:32:12,237 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=23, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase20.apache.org%2C33475%2C1689121928349.meta.1689121928922.meta, worker=jenkins-hbase20.apache.org,37887,1689121929610}] 2023-07-12 00:32:12,238 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 00:32:12,248 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 00:32:12,259 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689121932248,5,FailOnTimeoutGroup] 2023-07-12 00:32:12,260 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:12,260 INFO [M:0;jenkins-hbase20:42265] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,42265,1689121928263; zookeeper connection closed. 2023-07-12 00:32:12,260 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:42265-0x1007e974181001e, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:12,261 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689121932259,5,FailOnTimeoutGroup] 2023-07-12 00:32:12,261 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:12,261 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 00:32:12,261 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:12,261 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:12,262 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689121932261, completionTime=-1 2023-07-12 00:32:12,262 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-12 00:32:12,270 WARN [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1240): hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1689121932098, server=jenkins-hbase20.apache.org,33475,1689121928349}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2023-07-12 00:32:12,293 DEBUG [hconnection-0x3fc558ca-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase20.apache.org/148.251.75.209:33475 this server is in the failed servers list 2023-07-12 00:32:12,445 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,37887,1689121929610 2023-07-12 00:32:12,445 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 00:32:12,448 INFO [RS-EventLoopGroup-19-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57504, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 00:32:12,448 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=37887] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=24 2023-07-12 00:32:12,495 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(300): Splitting hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting/jenkins-hbase20.apache.org%2C33475%2C1689121928349.meta.1689121928922.meta, size=12.5 K (12790bytes) 2023-07-12 00:32:12,495 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting/jenkins-hbase20.apache.org%2C33475%2C1689121928349.meta.1689121928922.meta 2023-07-12 00:32:12,496 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting/jenkins-hbase20.apache.org%2C33475%2C1689121928349.meta.1689121928922.meta after 1ms 2023-07-12 00:32:12,500 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(311): Open hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting/jenkins-hbase20.apache.org%2C33475%2C1689121928349.meta.1689121928922.meta took 5ms 2023-07-12 00:32:12,555 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 2023-07-12 00:32:12,556 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting/jenkins-hbase20.apache.org%2C33475%2C1689121928349.meta.1689121928922.meta so closing down 2023-07-12 00:32:12,556 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-12 00:32:12,571 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0-Writer-1] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C33475%2C1689121928349.meta.1689121928922.meta.temp 2023-07-12 00:32:12,571 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-12 00:32:12,595 DEBUG [hconnection-0x3fc558ca-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase20.apache.org/148.251.75.209:33475 this server is in the failed servers list 2023-07-12 00:32:12,641 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-12 00:32:12,641 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-12 00:32:12,657 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C33475%2C1689121928349.meta.1689121928922.meta.temp (wrote 20 edits, skipped 0 edits in 0 ms) 2023-07-12 00:32:12,659 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C33475%2C1689121928349.meta.1689121928922.meta.temp to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/0000000000000000023 2023-07-12 00:32:12,659 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(423): Processed 21 edits across 1 Regions in 119 ms; skipped=1; WAL=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting/jenkins-hbase20.apache.org%2C33475%2C1689121928349.meta.1689121928922.meta, size=12.5 K, length=12790, corrupted=false, cancelled=false 2023-07-12 00:32:12,659 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting/jenkins-hbase20.apache.org%2C33475%2C1689121928349.meta.1689121928922.meta, journal: Splitting hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting/jenkins-hbase20.apache.org%2C33475%2C1689121928349.meta.1689121928922.meta, size=12.5 K (12790bytes) at 1689121932495Finishing writing output for hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting/jenkins-hbase20.apache.org%2C33475%2C1689121928349.meta.1689121928922.meta so closing down at 1689121932556 (+61 ms)Creating recovered edits writer path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C33475%2C1689121928349.meta.1689121928922.meta.temp at 1689121932571 (+15 ms)3 split writer threads finished at 1689121932571Closed recovered edits writer path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C33475%2C1689121928349.meta.1689121928922.meta.temp (wrote 20 edits, skipped 0 edits in 0 ms) at 1689121932657 (+86 ms)Rename recovered edits hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C33475%2C1689121928349.meta.1689121928922.meta.temp to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/0000000000000000023 at 1689121932659 (+2 ms)Processed 21 edits across 1 Regions in 119 ms; skipped=1; WAL=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting/jenkins-hbase20.apache.org%2C33475%2C1689121928349.meta.1689121928922.meta, size=12.5 K, length=12790, corrupted=false, cancelled=false at 1689121932659 2023-07-12 00:32:12,659 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting/jenkins-hbase20.apache.org%2C33475%2C1689121928349.meta.1689121928922.meta 2023-07-12 00:32:12,659 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=24 2023-07-12 00:32:12,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45743] master.HMaster(3641): Remote procedure done, pid=24 2023-07-12 00:32:12,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45743] wal.WALSplitUtil(141): Moved hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting/jenkins-hbase20.apache.org%2C33475%2C1689121928349.meta.1689121928922.meta to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/oldWALs 2023-07-12 00:32:12,671 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=23 2023-07-12 00:32:12,671 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=23, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase20.apache.org%2C33475%2C1689121928349.meta.1689121928922.meta, worker=jenkins-hbase20.apache.org,37887,1689121929610 in 431 msec 2023-07-12 00:32:12,672 DEBUG [PEWorker-4] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase20.apache.org,37887,1689121929610 2023-07-12 00:32:12,678 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-12 00:32:12,678 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; SplitWALProcedure jenkins-hbase20.apache.org%2C33475%2C1689121928349.meta.1689121928922.meta, worker=jenkins-hbase20.apache.org,37887,1689121929610 in 466 msec 2023-07-12 00:32:12,680 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting dir is empty, no logs to split. 2023-07-12 00:32:12,680 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase20.apache.org,33475,1689121928349 WAL count=0, meta=true 2023-07-12 00:32:12,680 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase20.apache.org,33475,1689121928349 WAL splitting is done? wals=0, meta=true 2023-07-12 00:32:12,690 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-12 00:32:12,691 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-12 00:32:12,693 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-12 00:32:12,843 DEBUG [jenkins-hbase20:45743] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-12 00:32:12,844 DEBUG [jenkins-hbase20:45743] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-07-12 00:32:12,846 DEBUG [jenkins-hbase20:45743] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 00:32:12,846 DEBUG [jenkins-hbase20:45743] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 00:32:12,847 DEBUG [jenkins-hbase20:45743] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 00:32:12,848 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,42069,1689121928319, state=OPENING 2023-07-12 00:32:12,850 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 00:32:12,853 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 00:32:12,854 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,42069,1689121928319}] 2023-07-12 00:32:13,020 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:13,027 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 00:32:13,029 INFO [RS-EventLoopGroup-17-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48464, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 00:32:13,038 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-12 00:32:13,039 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:32:13,044 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C42069%2C1689121928319.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,42069,1689121928319, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/oldWALs, maxLogs=32 2023-07-12 00:32:13,069 DEBUG [RS-EventLoopGroup-19-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:32:13,070 DEBUG [RS-EventLoopGroup-19-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:32:13,100 WARN [ReadOnlyZKClient-127.0.0.1:58965@0x0d42fd21] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-12 00:32:13,100 DEBUG [hconnection-0x3fc558ca-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 00:32:13,116 INFO [RS-EventLoopGroup-17-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48478, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 00:32:13,117 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=42069] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Get size: 123 connection: 148.251.75.209:48478 deadline: 1689121993117, exception=org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region hbase:meta,,1 is opening on jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:13,139 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,42069,1689121928319/jenkins-hbase20.apache.org%2C42069%2C1689121928319.meta.1689121933045.meta 2023-07-12 00:32:13,141 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK], DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK]] 2023-07-12 00:32:13,141 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-12 00:32:13,141 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-12 00:32:13,142 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-12 00:32:13,142 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-12 00:32:13,142 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-12 00:32:13,142 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:13,142 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-12 00:32:13,142 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-12 00:32:13,151 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 00:32:13,152 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/info 2023-07-12 00:32:13,152 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/info 2023-07-12 00:32:13,154 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 00:32:13,155 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:13,155 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 00:32:13,158 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/rep_barrier 2023-07-12 00:32:13,159 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/rep_barrier 2023-07-12 00:32:13,160 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 00:32:13,164 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:13,165 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 00:32:13,167 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/table 2023-07-12 00:32:13,168 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/table 2023-07-12 00:32:13,168 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 00:32:13,169 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:13,171 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740 2023-07-12 00:32:13,175 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740 2023-07-12 00:32:13,177 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/0000000000000000023 2023-07-12 00:32:13,186 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5464): Applied 74, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=23, path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/0000000000000000023 2023-07-12 00:32:13,187 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=11.17 KB heapSize=17.34 KB 2023-07-12 00:32:13,247 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=10.92 KB at sequenceid=23 (bloomFilter=false), to=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/.tmp/info/f9ecc69af1644af89b09389c38976702 2023-07-12 00:32:13,271 WARN [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1240): hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPENING, ts=1689121932853, server=jenkins-hbase20.apache.org,42069,1689121928319}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2023-07-12 00:32:13,299 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=254 B at sequenceid=23 (bloomFilter=false), to=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/.tmp/table/966cdcac31c54b609f3148c28ac881e4 2023-07-12 00:32:13,312 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/.tmp/info/f9ecc69af1644af89b09389c38976702 as hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/info/f9ecc69af1644af89b09389c38976702 2023-07-12 00:32:13,325 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/info/f9ecc69af1644af89b09389c38976702, entries=70, sequenceid=23, filesize=15.9 K 2023-07-12 00:32:13,327 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/.tmp/table/966cdcac31c54b609f3148c28ac881e4 as hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/table/966cdcac31c54b609f3148c28ac881e4 2023-07-12 00:32:13,339 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/table/966cdcac31c54b609f3148c28ac881e4, entries=4, sequenceid=23, filesize=4.9 K 2023-07-12 00:32:13,340 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~11.17 KB/11438, heapSize ~17.05 KB/17464, currentSize=0 B/0 for 1588230740 in 154ms, sequenceid=23, compaction requested=false; wal=null 2023-07-12 00:32:13,340 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-12 00:32:13,341 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/0000000000000000023 2023-07-12 00:32:13,346 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 00:32:13,350 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 00:32:13,355 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/23.seqid, newMaxSeqId=23, maxSeqId=1 2023-07-12 00:32:13,357 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=24; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65701091, jitterRate=-0.020977452397346497}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 00:32:13,357 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 00:32:13,358 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=26, masterSystemTime=1689121933020 2023-07-12 00:32:13,369 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-12 00:32:13,369 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-12 00:32:13,370 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,42069,1689121928319, state=OPEN 2023-07-12 00:32:13,371 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 00:32:13,372 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 00:32:13,379 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-07-12 00:32:13,379 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,42069,1689121928319 in 522 msec 2023-07-12 00:32:13,383 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=22 2023-07-12 00:32:13,383 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 693 msec 2023-07-12 00:32:14,154 WARN [RPCClient-NioEventLoopGroup-6-10] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase20.apache.org/148.251.75.209:33475 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:33475 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:32:14,157 DEBUG [RPCClient-NioEventLoopGroup-6-10] ipc.FailedServers(52): Added failed server with address jenkins-hbase20.apache.org/148.251.75.209:33475 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:33475 2023-07-12 00:32:14,368 DEBUG [hconnection-0x3fc558ca-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase20.apache.org/148.251.75.209:33475 this server is in the failed servers list 2023-07-12 00:32:14,443 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 00:32:14,630 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testLogSplittingAfterMasterRecoveryDueToZKExpiry' 2023-07-12 00:32:14,631 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-12 00:32:14,676 DEBUG [hconnection-0x3fc558ca-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase20.apache.org/148.251.75.209:33475 this server is in the failed servers list 2023-07-12 00:32:15,182 DEBUG [hconnection-0x3fc558ca-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase20.apache.org/148.251.75.209:33475 this server is in the failed servers list 2023-07-12 00:32:15,271 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 00:32:15,275 DEBUG [hconnection-0x1646f000-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 00:32:15,277 INFO [RS-EventLoopGroup-17-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48482, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 00:32:15,282 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=96bd68d39166bf0c53ccc701580614f1, regionState=OPEN, lastHost=jenkins-hbase20.apache.org,42069,1689121928319, regionLocation=jenkins-hbase20.apache.org,42069,1689121928319, openSeqNum=2 2023-07-12 00:32:15,282 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=2cba8ed741e888a6ec6550e0722fbc39, regionState=OPEN, lastHost=jenkins-hbase20.apache.org,33475,1689121928349, regionLocation=jenkins-hbase20.apache.org,33475,1689121928349, openSeqNum=2 2023-07-12 00:32:15,283 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=977900e03dd851ff67f68c586e17ff0e, regionState=OPEN, lastHost=jenkins-hbase20.apache.org,37887,1689121929610, regionLocation=jenkins-hbase20.apache.org,37887,1689121929610, openSeqNum=2 2023-07-12 00:32:15,283 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=810f496a1e86b8d9ebefb6030254ac4d, regionState=OPEN, lastHost=jenkins-hbase20.apache.org,42069,1689121928319, regionLocation=jenkins-hbase20.apache.org,42069,1689121928319, openSeqNum=2 2023-07-12 00:32:15,283 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=02432dc8a0cc182e04b7721d9cac70d0, regionState=OPEN, lastHost=jenkins-hbase20.apache.org,37887,1689121929610, regionLocation=jenkins-hbase20.apache.org,37887,1689121929610, openSeqNum=2 2023-07-12 00:32:15,283 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=e05a790448a127b53382088bce1a74d6, regionState=OPEN, lastHost=jenkins-hbase20.apache.org,33475,1689121928349, regionLocation=jenkins-hbase20.apache.org,33475,1689121928349, openSeqNum=2 2023-07-12 00:32:15,283 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=38818f72237f4c84213ef9820649179c, regionState=OPEN, lastHost=jenkins-hbase20.apache.org,42069,1689121928319, regionLocation=jenkins-hbase20.apache.org,42069,1689121928319, openSeqNum=2 2023-07-12 00:32:15,284 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-12 00:32:15,284 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689121995284 2023-07-12 00:32:15,284 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689122055284 2023-07-12 00:32:15,284 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 12 msec 2023-07-12 00:32:15,295 INFO [PEWorker-1] procedure.ServerCrashProcedure(199): jenkins-hbase20.apache.org,33475,1689121928349 had 3 regions 2023-07-12 00:32:15,296 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,45743,1689121928288-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:15,296 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,45743,1689121928288-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:15,296 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,45743,1689121928288-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:15,296 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:45743, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:15,296 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:15,297 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-07-12 00:32:15,300 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=22, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase20.apache.org,33475,1689121928349, splitWal=true, meta=true, isMeta: false 2023-07-12 00:32:15,302 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase20.apache.org,33475,1689121928349 WAL count=1, meta=false 2023-07-12 00:32:15,302 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=22, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase20.apache.org%2C33475%2C1689121928349.1689121928769}] 2023-07-12 00:32:15,306 WARN [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(172): unknown_server=jenkins-hbase20.apache.org,33475,1689121928349/testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39., unknown_server=jenkins-hbase20.apache.org,33475,1689121928349/testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6. 2023-07-12 00:32:15,306 DEBUG [PEWorker-3] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:15,310 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=27, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase20.apache.org%2C33475%2C1689121928349.1689121928769, worker=jenkins-hbase20.apache.org,42069,1689121928319}] 2023-07-12 00:32:15,312 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 00:32:15,314 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 00:32:15,314 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.449sec 2023-07-12 00:32:15,318 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 00:32:15,318 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 00:32:15,319 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 00:32:15,319 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,45743,1689121928288-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 00:32:15,319 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,45743,1689121928288-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 00:32:15,333 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 00:32:15,463 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=42069] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=28 2023-07-12 00:32:15,477 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(300): Splitting hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting/jenkins-hbase20.apache.org%2C33475%2C1689121928349.1689121928769, size=1.7 K (1708bytes) 2023-07-12 00:32:15,477 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting/jenkins-hbase20.apache.org%2C33475%2C1689121928349.1689121928769 2023-07-12 00:32:15,477 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting/jenkins-hbase20.apache.org%2C33475%2C1689121928349.1689121928769 after 0ms 2023-07-12 00:32:15,480 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(311): Open hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting/jenkins-hbase20.apache.org%2C33475%2C1689121928349.1689121928769 took 4ms 2023-07-12 00:32:15,483 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(346): Last flushed sequenceid for 2cba8ed741e888a6ec6550e0722fbc39: last_flushed_sequence_id: 18446744073709551615 2023-07-12 00:32:15,484 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(346): Last flushed sequenceid for e05a790448a127b53382088bce1a74d6: last_flushed_sequence_id: 18446744073709551615 2023-07-12 00:32:15,484 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting/jenkins-hbase20.apache.org%2C33475%2C1689121928349.1689121928769 so closing down 2023-07-12 00:32:15,484 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-12 00:32:15,487 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C33475%2C1689121928349.1689121928769.temp 2023-07-12 00:32:15,488 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-12 00:32:15,495 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C33475%2C1689121928349.1689121928769.temp (wrote 6 edits, skipped 0 edits in 0 ms) 2023-07-12 00:32:15,497 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C33475%2C1689121928349.1689121928769.temp to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39/recovered.edits/0000000000000000009 2023-07-12 00:32:15,497 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(423): Processed 8 edits across 1 Regions in 17 ms; skipped=2; WAL=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting/jenkins-hbase20.apache.org%2C33475%2C1689121928349.1689121928769, size=1.7 K, length=1708, corrupted=false, cancelled=false 2023-07-12 00:32:15,497 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting/jenkins-hbase20.apache.org%2C33475%2C1689121928349.1689121928769, journal: Splitting hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting/jenkins-hbase20.apache.org%2C33475%2C1689121928349.1689121928769, size=1.7 K (1708bytes) at 1689121935477Finishing writing output for hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting/jenkins-hbase20.apache.org%2C33475%2C1689121928349.1689121928769 so closing down at 1689121935484 (+7 ms)Creating recovered edits writer path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C33475%2C1689121928349.1689121928769.temp at 1689121935488 (+4 ms)3 split writer threads finished at 1689121935488Closed recovered edits writer path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C33475%2C1689121928349.1689121928769.temp (wrote 6 edits, skipped 0 edits in 0 ms) at 1689121935495 (+7 ms)Rename recovered edits hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C33475%2C1689121928349.1689121928769.temp to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39/recovered.edits/0000000000000000009 at 1689121935497 (+2 ms)Processed 8 edits across 1 Regions in 17 ms; skipped=2; WAL=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting/jenkins-hbase20.apache.org%2C33475%2C1689121928349.1689121928769, size=1.7 K, length=1708, corrupted=false, cancelled=false at 1689121935497 2023-07-12 00:32:15,497 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting/jenkins-hbase20.apache.org%2C33475%2C1689121928349.1689121928769 2023-07-12 00:32:15,497 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=28 2023-07-12 00:32:15,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45743] master.HMaster(3641): Remote procedure done, pid=28 2023-07-12 00:32:15,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45743] wal.WALSplitUtil(141): Moved hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting/jenkins-hbase20.apache.org%2C33475%2C1689121928349.1689121928769 to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/oldWALs 2023-07-12 00:32:15,501 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=28, resume processing ppid=27 2023-07-12 00:32:15,502 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=27, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase20.apache.org%2C33475%2C1689121928349.1689121928769, worker=jenkins-hbase20.apache.org,42069,1689121928319 in 190 msec 2023-07-12 00:32:15,502 DEBUG [PEWorker-2] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:15,504 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=22 2023-07-12 00:32:15,504 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=22, state=SUCCESS; SplitWALProcedure jenkins-hbase20.apache.org%2C33475%2C1689121928349.1689121928769, worker=jenkins-hbase20.apache.org,42069,1689121928319 in 200 msec 2023-07-12 00:32:15,506 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33475,1689121928349-splitting dir is empty, no logs to split. 2023-07-12 00:32:15,506 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase20.apache.org,33475,1689121928349 WAL count=0, meta=false 2023-07-12 00:32:15,506 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(290): Check if jenkins-hbase20.apache.org,33475,1689121928349 WAL splitting is done? wals=0, meta=false 2023-07-12 00:32:15,507 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=e05a790448a127b53382088bce1a74d6, ASSIGN}, {pid=30, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=2cba8ed741e888a6ec6550e0722fbc39, ASSIGN}] 2023-07-12 00:32:15,508 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=e05a790448a127b53382088bce1a74d6, ASSIGN 2023-07-12 00:32:15,508 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=30, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=2cba8ed741e888a6ec6550e0722fbc39, ASSIGN 2023-07-12 00:32:15,509 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=30, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=2cba8ed741e888a6ec6550e0722fbc39, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-12 00:32:15,509 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=e05a790448a127b53382088bce1a74d6, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-12 00:32:15,659 DEBUG [jenkins-hbase20:45743] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-07-12 00:32:15,660 DEBUG [jenkins-hbase20:45743] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 00:32:15,660 DEBUG [jenkins-hbase20:45743] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 00:32:15,660 DEBUG [jenkins-hbase20:45743] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 00:32:15,663 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=e05a790448a127b53382088bce1a74d6, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,37887,1689121929610 2023-07-12 00:32:15,663 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=2cba8ed741e888a6ec6550e0722fbc39, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:15,663 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6.","families":{"info":[{"qualifier":"regioninfo","vlen":84,"tag":[],"timestamp":"1689121935663"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121935663"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121935663"}]},"ts":"1689121935663"} 2023-07-12 00:32:15,663 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39.","families":{"info":[{"qualifier":"regioninfo","vlen":83,"tag":[],"timestamp":"1689121935663"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121935663"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121935663"}]},"ts":"1689121935663"} 2023-07-12 00:32:15,665 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=29, state=RUNNABLE; OpenRegionProcedure e05a790448a127b53382088bce1a74d6, server=jenkins-hbase20.apache.org,37887,1689121929610}] 2023-07-12 00:32:15,666 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=30, state=RUNNABLE; OpenRegionProcedure 2cba8ed741e888a6ec6550e0722fbc39, server=jenkins-hbase20.apache.org,42069,1689121928319}] 2023-07-12 00:32:15,820 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39. 2023-07-12 00:32:15,820 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6. 2023-07-12 00:32:15,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2cba8ed741e888a6ec6550e0722fbc39, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39.', STARTKEY => '', ENDKEY => '1'} 2023-07-12 00:32:15,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e05a790448a127b53382088bce1a74d6, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6.', STARTKEY => '4', ENDKEY => '5'} 2023-07-12 00:32:15,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testLogSplittingAfterMasterRecoveryDueToZKExpiry 2cba8ed741e888a6ec6550e0722fbc39 2023-07-12 00:32:15,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testLogSplittingAfterMasterRecoveryDueToZKExpiry e05a790448a127b53382088bce1a74d6 2023-07-12 00:32:15,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:15,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:15,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 2cba8ed741e888a6ec6550e0722fbc39 2023-07-12 00:32:15,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for e05a790448a127b53382088bce1a74d6 2023-07-12 00:32:15,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 2cba8ed741e888a6ec6550e0722fbc39 2023-07-12 00:32:15,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for e05a790448a127b53382088bce1a74d6 2023-07-12 00:32:15,822 INFO [StoreOpener-2cba8ed741e888a6ec6550e0722fbc39-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family col of region 2cba8ed741e888a6ec6550e0722fbc39 2023-07-12 00:32:15,822 INFO [StoreOpener-e05a790448a127b53382088bce1a74d6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family col of region e05a790448a127b53382088bce1a74d6 2023-07-12 00:32:15,823 DEBUG [StoreOpener-2cba8ed741e888a6ec6550e0722fbc39-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39/col 2023-07-12 00:32:15,823 DEBUG [StoreOpener-e05a790448a127b53382088bce1a74d6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/e05a790448a127b53382088bce1a74d6/col 2023-07-12 00:32:15,823 DEBUG [StoreOpener-e05a790448a127b53382088bce1a74d6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/e05a790448a127b53382088bce1a74d6/col 2023-07-12 00:32:15,823 DEBUG [StoreOpener-2cba8ed741e888a6ec6550e0722fbc39-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39/col 2023-07-12 00:32:15,823 INFO [StoreOpener-e05a790448a127b53382088bce1a74d6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e05a790448a127b53382088bce1a74d6 columnFamilyName col 2023-07-12 00:32:15,824 INFO [StoreOpener-2cba8ed741e888a6ec6550e0722fbc39-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2cba8ed741e888a6ec6550e0722fbc39 columnFamilyName col 2023-07-12 00:32:15,824 INFO [StoreOpener-e05a790448a127b53382088bce1a74d6-1] regionserver.HStore(310): Store=e05a790448a127b53382088bce1a74d6/col, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:15,824 INFO [StoreOpener-2cba8ed741e888a6ec6550e0722fbc39-1] regionserver.HStore(310): Store=2cba8ed741e888a6ec6550e0722fbc39/col, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:15,825 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/e05a790448a127b53382088bce1a74d6 2023-07-12 00:32:15,825 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39 2023-07-12 00:32:15,826 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/e05a790448a127b53382088bce1a74d6 2023-07-12 00:32:15,826 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39 2023-07-12 00:32:15,827 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39/recovered.edits/0000000000000000009 2023-07-12 00:32:15,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for e05a790448a127b53382088bce1a74d6 2023-07-12 00:32:15,829 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened e05a790448a127b53382088bce1a74d6; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67715156, jitterRate=0.009034454822540283}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:15,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for e05a790448a127b53382088bce1a74d6: 2023-07-12 00:32:15,830 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5464): Applied 6, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=9, path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39/recovered.edits/0000000000000000009 2023-07-12 00:32:15,830 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 2cba8ed741e888a6ec6550e0722fbc39 1/1 column families, dataSize=210 B heapSize=928 B 2023-07-12 00:32:15,831 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6., pid=31, masterSystemTime=1689121935817 2023-07-12 00:32:15,833 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6. 2023-07-12 00:32:15,833 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6. 2023-07-12 00:32:15,833 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=e05a790448a127b53382088bce1a74d6, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,37887,1689121929610 2023-07-12 00:32:15,833 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6.","families":{"info":[{"qualifier":"regioninfo","vlen":84,"tag":[],"timestamp":"1689121935833"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121935833"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121935833"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121935833"}]},"ts":"1689121935833"} 2023-07-12 00:32:15,837 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=29 2023-07-12 00:32:15,837 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=29, state=SUCCESS; OpenRegionProcedure e05a790448a127b53382088bce1a74d6, server=jenkins-hbase20.apache.org,37887,1689121929610 in 170 msec 2023-07-12 00:32:15,838 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=e05a790448a127b53382088bce1a74d6, ASSIGN in 330 msec 2023-07-12 00:32:15,840 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=210 B at sequenceid=9 (bloomFilter=true), to=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39/.tmp/col/136956e02e554924aa0409418feb8d2c 2023-07-12 00:32:15,845 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39/.tmp/col/136956e02e554924aa0409418feb8d2c as hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39/col/136956e02e554924aa0409418feb8d2c 2023-07-12 00:32:15,850 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39/col/136956e02e554924aa0409418feb8d2c, entries=6, sequenceid=9, filesize=4.9 K 2023-07-12 00:32:15,851 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~210 B/210, heapSize ~912 B/912, currentSize=0 B/0 for 2cba8ed741e888a6ec6550e0722fbc39 in 21ms, sequenceid=9, compaction requested=false; wal=null 2023-07-12 00:32:15,853 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39/recovered.edits/0000000000000000009 2023-07-12 00:32:15,856 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 2cba8ed741e888a6ec6550e0722fbc39 2023-07-12 00:32:15,858 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2cba8ed741e888a6ec6550e0722fbc39/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-07-12 00:32:15,858 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 2cba8ed741e888a6ec6550e0722fbc39; next sequenceid=10; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67350436, jitterRate=0.003599703311920166}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:15,858 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 2cba8ed741e888a6ec6550e0722fbc39: 2023-07-12 00:32:15,859 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39., pid=32, masterSystemTime=1689121935817 2023-07-12 00:32:15,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39. 2023-07-12 00:32:15,860 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39. 2023-07-12 00:32:15,861 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=2cba8ed741e888a6ec6550e0722fbc39, regionState=OPEN, openSeqNum=10, regionLocation=jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:15,861 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39.","families":{"info":[{"qualifier":"regioninfo","vlen":83,"tag":[],"timestamp":"1689121935861"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121935861"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121935861"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121935861"}]},"ts":"1689121935861"} 2023-07-12 00:32:15,864 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=30 2023-07-12 00:32:15,864 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=30, state=SUCCESS; OpenRegionProcedure 2cba8ed741e888a6ec6550e0722fbc39, server=jenkins-hbase20.apache.org,42069,1689121928319 in 196 msec 2023-07-12 00:32:15,865 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=22 2023-07-12 00:32:15,865 INFO [PEWorker-5] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase20.apache.org,33475,1689121928349 after splitting done 2023-07-12 00:32:15,866 DEBUG [PEWorker-5] master.DeadServer(114): Removed jenkins-hbase20.apache.org,33475,1689121928349 from processing; numProcessing=0 2023-07-12 00:32:15,866 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=2cba8ed741e888a6ec6550e0722fbc39, ASSIGN in 357 msec 2023-07-12 00:32:15,867 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=22, state=SUCCESS; ServerCrashProcedure jenkins-hbase20.apache.org,33475,1689121928349, splitWal=true, meta=true in 3.7520 sec 2023-07-12 00:32:16,200 DEBUG [Listener at localhost.localdomain/34481] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-07-12 00:32:16,205 DEBUG [hconnection-0x3fc558ca-shared-pool-3] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 00:32:16,207 INFO [RS-EventLoopGroup-19-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57508, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 00:32:16,208 DEBUG [Listener at localhost.localdomain/34481] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-07-12 00:32:16,213 DEBUG [Listener at localhost.localdomain/34481] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-07-12 00:32:16,217 DEBUG [Listener at localhost.localdomain/34481] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-07-12 00:32:16,221 DEBUG [Listener at localhost.localdomain/34481] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-07-12 00:32:16,226 DEBUG [Listener at localhost.localdomain/34481] master.HMaster(2711): Abort called but aborted=true, stopped=true 2023-07-12 00:32:16,226 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 00:32:16,226 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): ***** ABORTING master jenkins-hbase20.apache.org,45743,1689121928288: killAll ***** 2023-07-12 00:32:16,226 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,45743,1689121928288' ***** 2023-07-12 00:32:16,226 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2311): STOPPED: Stopped by Listener at localhost.localdomain/34481 2023-07-12 00:32:16,227 DEBUG [M:1;jenkins-hbase20:45743] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45e402f2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-12 00:32:16,227 INFO [M:1;jenkins-hbase20:45743] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,45743,1689121928288 2023-07-12 00:32:16,227 DEBUG [M:1;jenkins-hbase20:45743] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x18bc20ee to 127.0.0.1:58965 2023-07-12 00:32:16,227 DEBUG [M:1;jenkins-hbase20:45743] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:16,227 INFO [M:1;jenkins-hbase20:45743] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,45743,1689121928288; all regions closed. 2023-07-12 00:32:16,227 DEBUG [M:1;jenkins-hbase20:45743] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:16,227 DEBUG [M:1;jenkins-hbase20:45743] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 00:32:16,227 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 00:32:16,227 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689121932259] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689121932259,5,FailOnTimeoutGroup] 2023-07-12 00:32:16,227 DEBUG [M:1;jenkins-hbase20:45743] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 00:32:16,227 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689121932248] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689121932248,5,FailOnTimeoutGroup] 2023-07-12 00:32:16,229 INFO [M:1;jenkins-hbase20:45743] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 00:32:16,230 INFO [M:1;jenkins-hbase20:45743] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 00:32:16,230 INFO [M:1;jenkins-hbase20:45743] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-07-12 00:32:16,230 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:16,230 DEBUG [M:1;jenkins-hbase20:45743] master.HMaster(1512): Stopping service threads 2023-07-12 00:32:16,230 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:37887-0x1007e9741810026, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:16,230 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:16,230 INFO [M:1;jenkins-hbase20:45743] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 00:32:16,230 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:16,231 INFO [M:1;jenkins-hbase20:45743] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 00:32:16,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:16,230 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase20.apache.org,42069,1689121928319: killAll ***** 2023-07-12 00:32:16,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37887-0x1007e9741810026, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:16,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:16,234 ERROR [M:1;jenkins-hbase20:45743] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 00:32:16,234 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 00:32:16,234 INFO [M:1;jenkins-hbase20:45743] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=true 2023-07-12 00:32:16,234 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-12 00:32:16,234 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 00:32:16,235 DEBUG [M:1;jenkins-hbase20:45743] zookeeper.ZKUtil(398): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 00:32:16,235 WARN [M:1;jenkins-hbase20:45743] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 00:32:16,235 INFO [M:1;jenkins-hbase20:45743] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 00:32:16,236 INFO [M:1;jenkins-hbase20:45743] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=true 2023-07-12 00:32:16,237 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-12 00:32:16,239 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-12 00:32:16,240 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-12 00:32:16,243 DEBUG [M:1;jenkins-hbase20:45743] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 00:32:16,243 INFO [M:1;jenkins-hbase20:45743] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:16,243 DEBUG [M:1;jenkins-hbase20:45743] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:16,243 DEBUG [M:1;jenkins-hbase20:45743] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 00:32:16,243 DEBUG [M:1;jenkins-hbase20:45743] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:16,245 ERROR [M:1;jenkins-hbase20:45743] regionserver.HRegion(1825): Memstore data size is 41116 in region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:16,245 INFO [M:1;jenkins-hbase20:45743] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:16,245 DEBUG [M:1;jenkins-hbase20:45743] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 00:32:16,245 INFO [M:1;jenkins-hbase20:45743] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 00:32:16,245 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 00:32:16,245 INFO [M:1;jenkins-hbase20:45743] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:45743 2023-07-12 00:32:16,246 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1192755200, "init": 524288000, "max": 2051014656, "used": 430251392 }, "NonHeapMemoryUsage": { "committed": 141582336, "init": 2555904, "max": -1, "used": 139095416 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 0, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 45102, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 1, "ProcessCallTime_num_ops": 167, "ProcessCallTime_min": 0, "ProcessCallTime_max": 23, "ProcessCallTime_mean": 2, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 1, "ProcessCallTime_75th_percentile": 3, "ProcessCallTime_90th_percentile": 4, "ProcessCallTime_95th_percentile": 9, "ProcessCallTime_98th_percentile": 17, "ProcessCallTime_99th_percentile": 23, "ProcessCallTime_99.9th_percentile": 23, "ProcessCallTime_TimeRangeCount_0-1": 30, "ProcessCallTime_TimeRangeCount_1-3": 12, "ProcessCallTime_TimeRangeCount_3-10": 9, "ProcessCallTime_TimeRangeCount_10-30": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 167, "QueueCallTime_min": 0, "QueueCallTime_max": 6, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 1, "QueueCallTime_95th_percentile": 2, "QueueCallTime_98th_percentile": 3, "QueueCallTime_99th_percentile": 6, "QueueCallTime_99.9th_percentile": 6, "QueueCallTime_TimeRangeCount_0-1": 49, "QueueCallTime_TimeRangeCount_1-3": 3, "QueueCallTime_TimeRangeCount_3-10": 1, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 167, "TotalCallTime_min": 0, "TotalCallTime_max": 23, "TotalCallTime_mean": 2, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 1, "TotalCallTime_75th_percentile": 4, "TotalCallTime_90th_percentile": 7, "TotalCallTime_95th_percentile": 9, "TotalCallTime_98th_percentile": 17, "TotalCallTime_99th_percentile": 23, "TotalCallTime_99.9th_percentile": 23, "TotalCallTime_TimeRangeCount_0-1": 27, "TotalCallTime_TimeRangeCount_1-3": 12, "TotalCallTime_TimeRangeCount_3-10": 12, "TotalCallTime_TimeRangeCount_10-30": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 167, "ResponseSize_min": 0, "ResponseSize_max": 5990, "ResponseSize_mean": 208, "ResponseSize_25th_percentile": 1, "ResponseSize_median": 2, "ResponseSize_75th_percentile": 5, "ResponseSize_90th_percentile": 769, "ResponseSize_95th_percentile": 2683, "ResponseSize_98th_percentile": 4667, "ResponseSize_99th_percentile": 5328, "ResponseSize_99.9th_percentile": 5923, "ResponseSize_SizeRangeCount_0-10": 43, "ResponseSize_SizeRangeCount_10-100": 3, "ResponseSize_SizeRangeCount_100-1000": 7, "exceptions.UnknownScannerException": 0, "exceptions": 2, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 1, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 167, "RequestSize_min": 80, "RequestSize_max": 2899, "RequestSize_mean": 269, "RequestSize_25th_percentile": 145, "RequestSize_median": 176, "RequestSize_75th_percentile": 339, "RequestSize_90th_percentile": 1009, "RequestSize_95th_percentile": 1954, "RequestSize_98th_percentile": 2521, "RequestSize_99th_percentile": 2710, "RequestSize_99.9th_percentile": 2880, "RequestSize_SizeRangeCount_10-100": 5, "RequestSize_SizeRangeCount_100-1000": 48, "sentBytes": 28192 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:58965", "tag.serverName": "jenkins-hbase20.apache.org,42069,1689121928319", "tag.clusterId": "b855a1de-f551-45f5-8b9b-f7b8ff57b75f", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "regionCount": 4, "storeCount": 6, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 2, "maxStoreFileCount": 1, "memStoreSize": 78, "storeFileSize": 21223, "maxStoreFileAge": 385, "minStoreFileAge": 333, "avgStoreFileAge": 359, "numReferenceFiles": 0, "regionServerStartTime": 1689121928319, "averageRegionSize": 5325, "storeFileIndexSize": 920, "staticIndexSize": 290, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 79133, "localBytesRead": 79133, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819794032, "blockCacheCount": 2, "blockCacheDataBlockCount": 2, "blockCacheSize": 611856, "blockCacheCountHitPercent": 50.0, "blockCacheExpressHitPercent": 50.0, "l1CacheSize": 611856, "l1CacheFreeSize": 819794032, "l1CacheCount": 2, "l1CacheEvictionCount": 0, "l1CacheHitCount": 2, "l1CacheMissCount": 2, "l1CacheHitRatio": 0.5, "l1CacheMissRatio": 0.5, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 1.2, "writeRequestRatePerSecond": 0.4, "ByteBuffAllocatorHeapAllocationBytes": 147754, "ByteBuffAllocatorPoolAllocationBytes": 599040, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 12, "totalRowActionRequestCount": 8, "readRequestCount": 6, "filteredReadRequestCount": 0, "writeRequestCount": 2, "rpcGetRequestCount": 7, "rpcFullScanRequestCount": 0, "rpcScanRequestCount": 3, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 2, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 2, "blockCacheHitCountPrimary": 2, "blockCacheHitCachingCount": 2, "blockCacheMissCount": 2, "blockCacheMissCountPrimary": 2, "blockCacheMissCachingCount": 2, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 2, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 2, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 74, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 11438, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 3, "ScanTime_min": 0, "ScanTime_max": 0, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 0, "ScanTime_75th_percentile": 0, "ScanTime_90th_percentile": 0, "ScanTime_95th_percentile": 0, "ScanTime_98th_percentile": 0, "ScanTime_99th_percentile": 0, "ScanTime_99.9th_percentile": 0, "ScanTime_TimeRangeCount_0-1": 3, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 2, "Put_min": 1, "Put_max": 3, "Put_mean": 2, "Put_25th_percentile": 1, "Put_median": 2, "Put_75th_percentile": 3, "Put_90th_percentile": 3, "Put_95th_percentile": 3, "Put_98th_percentile": 3, "Put_99th_percentile": 3, "Put_99.9th_percentile": 3, "Put_TimeRangeCount_0-1": 1, "Put_TimeRangeCount_1-3": 1, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 1, "FlushMemstoreSize_min": 11438, "FlushMemstoreSize_max": 11438, "FlushMemstoreSize_mean": 11438, "FlushMemstoreSize_25th_percentile": 11438, "FlushMemstoreSize_median": 11438, "FlushMemstoreSize_75th_percentile": 11438, "FlushMemstoreSize_90th_percentile": 11438, "FlushMemstoreSize_95th_percentile": 11438, "FlushMemstoreSize_98th_percentile": 11438, "FlushMemstoreSize_99th_percentile": 11438, "FlushMemstoreSize_99.9th_percentile": 11438, "FlushMemstoreSize_SizeRangeCount_100-1000": 1, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 6, "Get_min": 0, "Get_max": 23, "Get_mean": 4, "Get_25th_percentile": 0, "Get_median": 1, "Get_75th_percentile": 2, "Get_90th_percentile": 23, "Get_95th_percentile": 23, "Get_98th_percentile": 23, "Get_99th_percentile": 23, "Get_99.9th_percentile": 23, "Get_TimeRangeCount_0-1": 4, "Get_TimeRangeCount_1-3": 1, "Get_TimeRangeCount_10-30": 1, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 21223, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 1, "FlushTime_min": 154, "FlushTime_max": 154, "FlushTime_mean": 154, "FlushTime_25th_percentile": 154, "FlushTime_median": 154, "FlushTime_75th_percentile": 154, "FlushTime_90th_percentile": 154, "FlushTime_95th_percentile": 154, "FlushTime_98th_percentile": 154, "FlushTime_99th_percentile": 154, "FlushTime_99.9th_percentile": 154, "FlushTime_TimeRangeCount_100-300": 1, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 3, "ScanSize_min": 86, "ScanSize_max": 985, "ScanSize_mean": 685, "ScanSize_25th_percentile": 86, "ScanSize_median": 438, "ScanSize_75th_percentile": 711, "ScanSize_90th_percentile": 875, "ScanSize_95th_percentile": 930, "ScanSize_98th_percentile": 963, "ScanSize_99th_percentile": 974, "ScanSize_99.9th_percentile": 983, "ScanSize_SizeRangeCount_10-100": 1, "ScanSize_SizeRangeCount_100-1000": 2, "slowGetCount": 0, "flushedMemstoreBytes": 11438, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 1, "FlushOutputSize_min": 21223, "FlushOutputSize_max": 21223, "FlushOutputSize_mean": 21223, "FlushOutputSize_25th_percentile": 21223, "FlushOutputSize_median": 21223, "FlushOutputSize_75th_percentile": 21223, "FlushOutputSize_90th_percentile": 21223, "FlushOutputSize_95th_percentile": 21223, "FlushOutputSize_98th_percentile": 21223, "FlushOutputSize_99th_percentile": 21223, "FlushOutputSize_99.9th_percentile": 21223, "FlushOutputSize_SizeRangeCount_100-1000": 1, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 93, "ServerReadQueryPerSecond_mean_rate": 3.0241271332370667, "ServerReadQueryPerSecond_1min_rate": 2.81760632426473, "ServerReadQueryPerSecond_5min_rate": 2.6383756459467356, "ServerReadQueryPerSecond_15min_rate": 2.6124191975460884, "ServerWriteQueryPerSecond_count": 74, "ServerWriteQueryPerSecond_mean_rate": 2.406294348075967, "ServerWriteQueryPerSecond_1min_rate": 1.902961977749651, "ServerWriteQueryPerSecond_5min_rate": 1.5054020168326836, "ServerWriteQueryPerSecond_15min_rate": 1.4354140700006486 } ] } 2023-07-12 00:32:16,247 WARN [RS-EventLoopGroup-18-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase20.apache.org/148.251.75.209:45743 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:45743 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:32:16,247 WARN [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2576): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.net.ConnectException: Call to address=jenkins-hbase20.apache.org/148.251.75.209:45743 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:45743 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:334) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:581) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:16481) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2573) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:196) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$200(MiniHBaseCluster.java:123) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:189) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:186) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2597) at org.apache.hadoop.hbase.MiniHBaseCluster.killAll(MiniHBaseCluster.java:913) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:108) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) Caused by: java.net.ConnectException: Call to address=jenkins-hbase20.apache.org/148.251.75.209:45743 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:45743 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:186) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:385) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:99) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:398) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:368) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1428) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:396) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:913) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:195) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$300(NettyRpcConnection.java:76) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:296) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:287) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:674) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:693) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:45743 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:32:16,247 DEBUG [M:1;jenkins-hbase20:45743] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,45743,1689121928288 already deleted, retry=false 2023-07-12 00:32:16,247 DEBUG [RS-EventLoopGroup-18-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase20.apache.org/148.251.75.209:45743 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:45743 2023-07-12 00:32:16,251 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,42069,1689121928319' ***** 2023-07-12 00:32:16,251 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2311): STOPPED: killAll 2023-07-12 00:32:16,253 DEBUG [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2541): Abort already in progress. Ignoring the current request with reason: killAll 2023-07-12 00:32:16,253 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase20.apache.org,37887,1689121929610: killAll ***** 2023-07-12 00:32:16,253 INFO [RS:0;jenkins-hbase20:42069] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 00:32:16,253 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 00:32:16,253 INFO [RS:0;jenkins-hbase20:42069] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-12 00:32:16,253 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-12 00:32:16,253 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 00:32:16,253 INFO [RS:0;jenkins-hbase20:42069] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-12 00:32:16,254 INFO [RS:0;jenkins-hbase20:42069] regionserver.HRegionServer(3305): Received CLOSE for 810f496a1e86b8d9ebefb6030254ac4d 2023-07-12 00:32:16,254 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-12 00:32:16,254 INFO [RS:0;jenkins-hbase20:42069] regionserver.HRegionServer(3305): Received CLOSE for 38818f72237f4c84213ef9820649179c 2023-07-12 00:32:16,255 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 810f496a1e86b8d9ebefb6030254ac4d, disabling compactions & flushes 2023-07-12 00:32:16,255 INFO [RS:0;jenkins-hbase20:42069] regionserver.HRegionServer(3305): Received CLOSE for 2cba8ed741e888a6ec6550e0722fbc39 2023-07-12 00:32:16,255 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1689121929714.810f496a1e86b8d9ebefb6030254ac4d. 2023-07-12 00:32:16,255 INFO [RS:0;jenkins-hbase20:42069] regionserver.HRegionServer(3305): Received CLOSE for 96bd68d39166bf0c53ccc701580614f1 2023-07-12 00:32:16,255 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1689121929714.810f496a1e86b8d9ebefb6030254ac4d. 2023-07-12 00:32:16,256 INFO [RS:0;jenkins-hbase20:42069] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:16,256 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1689121929714.810f496a1e86b8d9ebefb6030254ac4d. after waiting 0 ms 2023-07-12 00:32:16,256 DEBUG [RS:0;jenkins-hbase20:42069] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x176c4c2d to 127.0.0.1:58965 2023-07-12 00:32:16,256 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-12 00:32:16,256 DEBUG [RS:0;jenkins-hbase20:42069] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:16,256 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1689121929714.810f496a1e86b8d9ebefb6030254ac4d. 2023-07-12 00:32:16,256 INFO [RS:0;jenkins-hbase20:42069] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 00:32:16,256 INFO [RS:0;jenkins-hbase20:42069] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 00:32:16,256 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-12 00:32:16,256 INFO [RS:0;jenkins-hbase20:42069] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 00:32:16,256 INFO [RS:0;jenkins-hbase20:42069] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-12 00:32:16,257 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1192755200, "init": 524288000, "max": 2051014656, "used": 431342344 }, "NonHeapMemoryUsage": { "committed": 141647872, "init": 2555904, "max": -1, "used": 139139472 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 0, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 45102, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 1, "ProcessCallTime_num_ops": 167, "ProcessCallTime_min": 0, "ProcessCallTime_max": 23, "ProcessCallTime_mean": 2, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 1, "ProcessCallTime_75th_percentile": 3, "ProcessCallTime_90th_percentile": 4, "ProcessCallTime_95th_percentile": 9, "ProcessCallTime_98th_percentile": 17, "ProcessCallTime_99th_percentile": 23, "ProcessCallTime_99.9th_percentile": 23, "ProcessCallTime_TimeRangeCount_0-1": 30, "ProcessCallTime_TimeRangeCount_1-3": 12, "ProcessCallTime_TimeRangeCount_3-10": 9, "ProcessCallTime_TimeRangeCount_10-30": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 167, "QueueCallTime_min": 0, "QueueCallTime_max": 6, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 1, "QueueCallTime_95th_percentile": 2, "QueueCallTime_98th_percentile": 3, "QueueCallTime_99th_percentile": 6, "QueueCallTime_99.9th_percentile": 6, "QueueCallTime_TimeRangeCount_0-1": 49, "QueueCallTime_TimeRangeCount_1-3": 3, "QueueCallTime_TimeRangeCount_3-10": 1, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 167, "TotalCallTime_min": 0, "TotalCallTime_max": 23, "TotalCallTime_mean": 2, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 1, "TotalCallTime_75th_percentile": 4, "TotalCallTime_90th_percentile": 7, "TotalCallTime_95th_percentile": 9, "TotalCallTime_98th_percentile": 17, "TotalCallTime_99th_percentile": 23, "TotalCallTime_99.9th_percentile": 23, "TotalCallTime_TimeRangeCount_0-1": 27, "TotalCallTime_TimeRangeCount_1-3": 12, "TotalCallTime_TimeRangeCount_3-10": 12, "TotalCallTime_TimeRangeCount_10-30": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 167, "ResponseSize_min": 0, "ResponseSize_max": 5990, "ResponseSize_mean": 208, "ResponseSize_25th_percentile": 1, "ResponseSize_median": 2, "ResponseSize_75th_percentile": 5, "ResponseSize_90th_percentile": 769, "ResponseSize_95th_percentile": 2683, "ResponseSize_98th_percentile": 4667, "ResponseSize_99th_percentile": 5328, "ResponseSize_99.9th_percentile": 5923, "ResponseSize_SizeRangeCount_0-10": 43, "ResponseSize_SizeRangeCount_10-100": 3, "ResponseSize_SizeRangeCount_100-1000": 7, "exceptions.UnknownScannerException": 0, "exceptions": 2, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 1, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 167, "RequestSize_min": 80, "RequestSize_max": 2899, "RequestSize_mean": 269, "RequestSize_25th_percentile": 145, "RequestSize_median": 176, "RequestSize_75th_percentile": 339, "RequestSize_90th_percentile": 1009, "RequestSize_95th_percentile": 1954, "RequestSize_98th_percentile": 2521, "RequestSize_99th_percentile": 2710, "RequestSize_99.9th_percentile": 2880, "RequestSize_SizeRangeCount_10-100": 5, "RequestSize_SizeRangeCount_100-1000": 48, "sentBytes": 28192 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:58965", "tag.serverName": "jenkins-hbase20.apache.org,42069,1689121928319", "tag.clusterId": "b855a1de-f551-45f5-8b9b-f7b8ff57b75f", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "regionCount": 4, "storeCount": 6, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 2, "maxStoreFileCount": 1, "memStoreSize": 78, "storeFileSize": 21223, "maxStoreFileAge": 385, "minStoreFileAge": 333, "avgStoreFileAge": 359, "numReferenceFiles": 0, "regionServerStartTime": 1689121928319, "averageRegionSize": 5325, "storeFileIndexSize": 920, "staticIndexSize": 290, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 79133, "localBytesRead": 79133, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819794032, "blockCacheCount": 2, "blockCacheDataBlockCount": 2, "blockCacheSize": 611856, "blockCacheCountHitPercent": 50.0, "blockCacheExpressHitPercent": 50.0, "l1CacheSize": 611856, "l1CacheFreeSize": 819794032, "l1CacheCount": 2, "l1CacheEvictionCount": 0, "l1CacheHitCount": 2, "l1CacheMissCount": 2, "l1CacheHitRatio": 0.5, "l1CacheMissRatio": 0.5, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 1.2, "writeRequestRatePerSecond": 0.4, "ByteBuffAllocatorHeapAllocationBytes": 147754, "ByteBuffAllocatorPoolAllocationBytes": 599040, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 12, "totalRowActionRequestCount": 8, "readRequestCount": 6, "filteredReadRequestCount": 0, "writeRequestCount": 2, "rpcGetRequestCount": 7, "rpcFullScanRequestCount": 0, "rpcScanRequestCount": 3, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 2, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 2, "blockCacheHitCountPrimary": 2, "blockCacheHitCachingCount": 2, "blockCacheMissCount": 2, "blockCacheMissCountPrimary": 2, "blockCacheMissCachingCount": 2, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 2, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 2, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 74, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 11438, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 3, "ScanTime_min": 0, "ScanTime_max": 0, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 0, "ScanTime_75th_percentile": 0, "ScanTime_90th_percentile": 0, "ScanTime_95th_percentile": 0, "ScanTime_98th_percentile": 0, "ScanTime_99th_percentile": 0, "ScanTime_99.9th_percentile": 0, "ScanTime_TimeRangeCount_0-1": 3, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 2, "Put_min": 1, "Put_max": 3, "Put_mean": 2, "Put_25th_percentile": 1, "Put_median": 2, "Put_75th_percentile": 3, "Put_90th_percentile": 3, "Put_95th_percentile": 3, "Put_98th_percentile": 3, "Put_99th_percentile": 3, "Put_99.9th_percentile": 3, "Put_TimeRangeCount_0-1": 1, "Put_TimeRangeCount_1-3": 1, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 1, "FlushMemstoreSize_min": 11438, "FlushMemstoreSize_max": 11438, "FlushMemstoreSize_mean": 11438, "FlushMemstoreSize_25th_percentile": 11438, "FlushMemstoreSize_median": 11438, "FlushMemstoreSize_75th_percentile": 11438, "FlushMemstoreSize_90th_percentile": 11438, "FlushMemstoreSize_95th_percentile": 11438, "FlushMemstoreSize_98th_percentile": 11438, "FlushMemstoreSize_99th_percentile": 11438, "FlushMemstoreSize_99.9th_percentile": 11438, "FlushMemstoreSize_SizeRangeCount_100-1000": 1, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 6, "Get_min": 0, "Get_max": 23, "Get_mean": 4, "Get_25th_percentile": 0, "Get_median": 1, "Get_75th_percentile": 2, "Get_90th_percentile": 23, "Get_95th_percentile": 23, "Get_98th_percentile": 23, "Get_99th_percentile": 23, "Get_99.9th_percentile": 23, "Get_TimeRangeCount_0-1": 4, "Get_TimeRangeCount_1-3": 1, "Get_TimeRangeCount_10-30": 1, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 21223, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 1, "FlushTime_min": 154, "FlushTime_max": 154, "FlushTime_mean": 154, "FlushTime_25th_percentile": 154, "FlushTime_median": 154, "FlushTime_75th_percentile": 154, "FlushTime_90th_percentile": 154, "FlushTime_95th_percentile": 154, "FlushTime_98th_percentile": 154, "FlushTime_99th_percentile": 154, "FlushTime_99.9th_percentile": 154, "FlushTime_TimeRangeCount_100-300": 1, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 3, "ScanSize_min": 86, "ScanSize_max": 985, "ScanSize_mean": 685, "ScanSize_25th_percentile": 86, "ScanSize_median": 438, "ScanSize_75th_percentile": 711, "ScanSize_90th_percentile": 875, "ScanSize_95th_percentile": 930, "ScanSize_98th_percentile": 963, "ScanSize_99th_percentile": 974, "ScanSize_99.9th_percentile": 983, "ScanSize_SizeRangeCount_10-100": 1, "ScanSize_SizeRangeCount_100-1000": 2, "slowGetCount": 0, "flushedMemstoreBytes": 11438, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 1, "FlushOutputSize_min": 21223, "FlushOutputSize_max": 21223, "FlushOutputSize_mean": 21223, "FlushOutputSize_25th_percentile": 21223, "FlushOutputSize_median": 21223, "FlushOutputSize_75th_percentile": 21223, "FlushOutputSize_90th_percentile": 21223, "FlushOutputSize_95th_percentile": 21223, "FlushOutputSize_98th_percentile": 21223, "FlushOutputSize_99th_percentile": 21223, "FlushOutputSize_99.9th_percentile": 21223, "FlushOutputSize_SizeRangeCount_100-1000": 1, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 93, "ServerReadQueryPerSecond_mean_rate": 3.0241271332370667, "ServerReadQueryPerSecond_1min_rate": 2.81760632426473, "ServerReadQueryPerSecond_5min_rate": 2.6383756459467356, "ServerReadQueryPerSecond_15min_rate": 2.6124191975460884, "ServerWriteQueryPerSecond_count": 74, "ServerWriteQueryPerSecond_mean_rate": 2.406294348075967, "ServerWriteQueryPerSecond_1min_rate": 1.902961977749651, "ServerWriteQueryPerSecond_5min_rate": 1.5054020168326836, "ServerWriteQueryPerSecond_15min_rate": 1.4354140700006486 } ] } 2023-07-12 00:32:16,258 WARN [RS-EventLoopGroup-19-3] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase20.apache.org/148.251.75.209:45743 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:45743 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:32:16,258 WARN [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2576): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.net.ConnectException: Call to address=jenkins-hbase20.apache.org/148.251.75.209:45743 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:45743 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:334) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:581) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:16481) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2573) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:196) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$200(MiniHBaseCluster.java:123) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:189) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:186) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2597) at org.apache.hadoop.hbase.MiniHBaseCluster.killAll(MiniHBaseCluster.java:913) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:108) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) Caused by: java.net.ConnectException: Call to address=jenkins-hbase20.apache.org/148.251.75.209:45743 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:45743 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:186) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:385) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:99) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:398) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:368) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1428) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:396) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:913) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:195) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$300(NettyRpcConnection.java:76) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:296) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:287) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:674) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:693) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:45743 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:32:16,261 INFO [RS:0;jenkins-hbase20:42069] regionserver.HRegionServer(1474): Waiting on 5 regions to close 2023-07-12 00:32:16,259 DEBUG [RS-EventLoopGroup-19-3] ipc.FailedServers(52): Added failed server with address jenkins-hbase20.apache.org/148.251.75.209:45743 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:45743 2023-07-12 00:32:16,258 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 00:32:16,263 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,37887,1689121929610' ***** 2023-07-12 00:32:16,263 DEBUG [RS:0;jenkins-hbase20:42069] regionserver.HRegionServer(1478): Online Regions={810f496a1e86b8d9ebefb6030254ac4d=testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1689121929714.810f496a1e86b8d9ebefb6030254ac4d., 38818f72237f4c84213ef9820649179c=testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1689121929714.38818f72237f4c84213ef9820649179c., 1588230740=hbase:meta,,1.1588230740, 2cba8ed741e888a6ec6550e0722fbc39=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39., 96bd68d39166bf0c53ccc701580614f1=hbase:namespace,,1689121929001.96bd68d39166bf0c53ccc701580614f1.} 2023-07-12 00:32:16,263 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 00:32:16,265 DEBUG [RS:0;jenkins-hbase20:42069] regionserver.HRegionServer(1504): Waiting on 1588230740, 2cba8ed741e888a6ec6550e0722fbc39, 38818f72237f4c84213ef9820649179c, 810f496a1e86b8d9ebefb6030254ac4d, 96bd68d39166bf0c53ccc701580614f1 2023-07-12 00:32:16,261 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1689121929714.810f496a1e86b8d9ebefb6030254ac4d. 2023-07-12 00:32:16,265 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 00:32:16,265 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 00:32:16,265 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 00:32:16,265 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 00:32:16,265 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2311): STOPPED: killAll 2023-07-12 00:32:16,265 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 810f496a1e86b8d9ebefb6030254ac4d: 2023-07-12 00:32:16,266 INFO [Listener at localhost.localdomain/34481] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-12 00:32:16,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1689121929714.810f496a1e86b8d9ebefb6030254ac4d. 2023-07-12 00:32:16,266 INFO [RS:2;jenkins-hbase20:37887] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 00:32:16,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 38818f72237f4c84213ef9820649179c, disabling compactions & flushes 2023-07-12 00:32:16,266 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1689121929714.38818f72237f4c84213ef9820649179c. 2023-07-12 00:32:16,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1689121929714.38818f72237f4c84213ef9820649179c. 2023-07-12 00:32:16,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1689121929714.38818f72237f4c84213ef9820649179c. after waiting 0 ms 2023-07-12 00:32:16,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1689121929714.38818f72237f4c84213ef9820649179c. 2023-07-12 00:32:16,266 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 00:32:16,266 INFO [RS:2;jenkins-hbase20:37887] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-12 00:32:16,267 INFO [RS:2;jenkins-hbase20:37887] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-12 00:32:16,269 INFO [RS:2;jenkins-hbase20:37887] regionserver.HRegionServer(3305): Received CLOSE for e05a790448a127b53382088bce1a74d6 2023-07-12 00:32:16,269 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1689121929714.38818f72237f4c84213ef9820649179c. 2023-07-12 00:32:16,269 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 38818f72237f4c84213ef9820649179c: 2023-07-12 00:32:16,269 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0d42fd21 to 127.0.0.1:58965 2023-07-12 00:32:16,269 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1689121929714.38818f72237f4c84213ef9820649179c. 2023-07-12 00:32:16,270 DEBUG [Listener at localhost.localdomain/34481] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:16,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 2cba8ed741e888a6ec6550e0722fbc39, disabling compactions & flushes 2023-07-12 00:32:16,270 DEBUG [Listener at localhost.localdomain/34481] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-12 00:32:16,270 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39. 2023-07-12 00:32:16,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39. 2023-07-12 00:32:16,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39. after waiting 0 ms 2023-07-12 00:32:16,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39. 2023-07-12 00:32:16,271 INFO [RS:2;jenkins-hbase20:37887] regionserver.HRegionServer(3305): Received CLOSE for 977900e03dd851ff67f68c586e17ff0e 2023-07-12 00:32:16,271 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing e05a790448a127b53382088bce1a74d6, disabling compactions & flushes 2023-07-12 00:32:16,271 INFO [RS:2;jenkins-hbase20:37887] regionserver.HRegionServer(3305): Received CLOSE for 02432dc8a0cc182e04b7721d9cac70d0 2023-07-12 00:32:16,271 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6. 2023-07-12 00:32:16,271 INFO [RS:2;jenkins-hbase20:37887] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,37887,1689121929610 2023-07-12 00:32:16,271 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6. 2023-07-12 00:32:16,271 DEBUG [RS:2;jenkins-hbase20:37887] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4caa4cc4 to 127.0.0.1:58965 2023-07-12 00:32:16,271 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6. after waiting 0 ms 2023-07-12 00:32:16,271 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6. 2023-07-12 00:32:16,271 DEBUG [RS:2;jenkins-hbase20:37887] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:16,271 INFO [RS:2;jenkins-hbase20:37887] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-12 00:32:16,271 DEBUG [RS:2;jenkins-hbase20:37887] regionserver.HRegionServer(1478): Online Regions={e05a790448a127b53382088bce1a74d6=testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6., 977900e03dd851ff67f68c586e17ff0e=testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1689121929714.977900e03dd851ff67f68c586e17ff0e., 02432dc8a0cc182e04b7721d9cac70d0=testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1689121929714.02432dc8a0cc182e04b7721d9cac70d0.} 2023-07-12 00:32:16,271 DEBUG [RS:2;jenkins-hbase20:37887] regionserver.HRegionServer(1504): Waiting on 02432dc8a0cc182e04b7721d9cac70d0, 977900e03dd851ff67f68c586e17ff0e, e05a790448a127b53382088bce1a74d6 2023-07-12 00:32:16,278 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6. 2023-07-12 00:32:16,278 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1825): Memstore data size is 2626 in region hbase:meta,,1.1588230740 2023-07-12 00:32:16,279 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for e05a790448a127b53382088bce1a74d6: 2023-07-12 00:32:16,279 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 00:32:16,279 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1689121929714.e05a790448a127b53382088bce1a74d6. 2023-07-12 00:32:16,280 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 00:32:16,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 977900e03dd851ff67f68c586e17ff0e, disabling compactions & flushes 2023-07-12 00:32:16,281 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 00:32:16,282 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1689121929714.977900e03dd851ff67f68c586e17ff0e. 2023-07-12 00:32:16,282 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39. 2023-07-12 00:32:16,282 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-12 00:32:16,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 2cba8ed741e888a6ec6550e0722fbc39: 2023-07-12 00:32:16,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1689121929714.977900e03dd851ff67f68c586e17ff0e. 2023-07-12 00:32:16,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1689121929714.977900e03dd851ff67f68c586e17ff0e. after waiting 0 ms 2023-07-12 00:32:16,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1689121929714.2cba8ed741e888a6ec6550e0722fbc39. 2023-07-12 00:32:16,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1689121929714.977900e03dd851ff67f68c586e17ff0e. 2023-07-12 00:32:16,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 96bd68d39166bf0c53ccc701580614f1, disabling compactions & flushes 2023-07-12 00:32:16,282 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689121929001.96bd68d39166bf0c53ccc701580614f1. 2023-07-12 00:32:16,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689121929001.96bd68d39166bf0c53ccc701580614f1. 2023-07-12 00:32:16,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689121929001.96bd68d39166bf0c53ccc701580614f1. after waiting 0 ms 2023-07-12 00:32:16,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689121929001.96bd68d39166bf0c53ccc701580614f1. 2023-07-12 00:32:16,283 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1689121929714.977900e03dd851ff67f68c586e17ff0e. 2023-07-12 00:32:16,283 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 977900e03dd851ff67f68c586e17ff0e: 2023-07-12 00:32:16,283 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689121929001.96bd68d39166bf0c53ccc701580614f1. 2023-07-12 00:32:16,283 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1689121929714.977900e03dd851ff67f68c586e17ff0e. 2023-07-12 00:32:16,284 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689121929001.96bd68d39166bf0c53ccc701580614f1. 2023-07-12 00:32:16,284 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 96bd68d39166bf0c53ccc701580614f1: 2023-07-12 00:32:16,284 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 02432dc8a0cc182e04b7721d9cac70d0, disabling compactions & flushes 2023-07-12 00:32:16,284 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1689121929714.02432dc8a0cc182e04b7721d9cac70d0. 2023-07-12 00:32:16,284 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689121929001.96bd68d39166bf0c53ccc701580614f1. 2023-07-12 00:32:16,284 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1689121929714.02432dc8a0cc182e04b7721d9cac70d0. 2023-07-12 00:32:16,284 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1689121929714.02432dc8a0cc182e04b7721d9cac70d0. after waiting 0 ms 2023-07-12 00:32:16,284 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1689121929714.02432dc8a0cc182e04b7721d9cac70d0. 2023-07-12 00:32:16,285 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1689121929714.02432dc8a0cc182e04b7721d9cac70d0. 2023-07-12 00:32:16,285 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 02432dc8a0cc182e04b7721d9cac70d0: 2023-07-12 00:32:16,285 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1689121929714.02432dc8a0cc182e04b7721d9cac70d0. 2023-07-12 00:32:16,290 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 00:32:16,356 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:16,356 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45743-0x1007e974181001f, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:16,356 INFO [M:1;jenkins-hbase20:45743] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,45743,1689121928288; zookeeper connection closed. 2023-07-12 00:32:16,465 INFO [RS:0;jenkins-hbase20:42069] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,42069,1689121928319; all regions closed. 2023-07-12 00:32:16,471 INFO [RS:2;jenkins-hbase20:37887] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,37887,1689121929610; all regions closed. 2023-07-12 00:32:16,476 DEBUG [RS:0;jenkins-hbase20:42069] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:16,476 DEBUG [RS:2;jenkins-hbase20:37887] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:16,476 INFO [RS:0;jenkins-hbase20:42069] regionserver.LeaseManager(133): Closed leases 2023-07-12 00:32:16,476 INFO [RS:2;jenkins-hbase20:37887] regionserver.LeaseManager(133): Closed leases 2023-07-12 00:32:16,477 INFO [RS:0;jenkins-hbase20:42069] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-12 00:32:16,477 INFO [RS:2;jenkins-hbase20:37887] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-12 00:32:16,477 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 00:32:16,477 INFO [RS:2;jenkins-hbase20:37887] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 00:32:16,477 INFO [RS:2;jenkins-hbase20:37887] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 00:32:16,477 INFO [RS:2;jenkins-hbase20:37887] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 00:32:16,478 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 00:32:16,478 INFO [RS:0;jenkins-hbase20:42069] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:42069 2023-07-12 00:32:16,479 INFO [RS:2;jenkins-hbase20:37887] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:37887 2023-07-12 00:32:16,480 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:37887-0x1007e9741810026, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:16,480 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:37887-0x1007e9741810026, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:16,480 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,42069,1689121928319 2023-07-12 00:32:16,480 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:16,480 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,37887,1689121929610 2023-07-12 00:32:16,480 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:37887-0x1007e9741810026, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,37887,1689121929610 2023-07-12 00:32:16,763 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:37887-0x1007e9741810026, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:16,764 INFO [RS:2;jenkins-hbase20:37887] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,37887,1689121929610; zookeeper connection closed. 2023-07-12 00:32:16,764 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:37887-0x1007e9741810026, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:16,764 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5303360c] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5303360c 2023-07-12 00:32:16,864 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:16,864 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:42069-0x1007e9741810020, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:16,864 INFO [RS:0;jenkins-hbase20:42069] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,42069,1689121928319; zookeeper connection closed. 2023-07-12 00:32:16,864 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@1da6be64] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@1da6be64 2023-07-12 00:32:16,864 INFO [Listener at localhost.localdomain/34481] util.JVMClusterUtil(335): Shutdown of 2 master(s) and 3 regionserver(s) complete 2023-07-12 00:32:16,966 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e974181001d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:16,966 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:16,969 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=testing utility connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:32:16,971 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility0x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:32:16,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): testing utility-0x1007e9741810029 connected 2023-07-12 00:32:16,973 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2023-07-12 00:32:16,973 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs 2023-07-12 00:32:16,973 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/splitWAL 2023-07-12 00:32:16,974 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters 2023-07-12 00:32:16,974 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc 2023-07-12 00:32:16,974 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master-maintenance 2023-07-12 00:32:16,975 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot 2023-07-12 00:32:16,975 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/switch 2023-07-12 00:32:16,975 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-12 00:32:16,975 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/draining 2023-07-12 00:32:16,976 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-07-12 00:32:16,976 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/hbaseid 2023-07-12 00:32:16,976 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/table 2023-07-12 00:32:16,978 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/abort 2023-07-12 00:32:16,978 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired 2023-07-12 00:32:16,978 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached 2023-07-12 00:32:16,979 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/abort 2023-07-12 00:32:16,980 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired 2023-07-12 00:32:16,980 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached 2023-07-12 00:32:16,981 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/default 2023-07-12 00:32:16,981 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/hbase 2023-07-12 00:32:16,982 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/table/hbase:meta 2023-07-12 00:32:16,982 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/table/hbase:namespace 2023-07-12 00:32:16,983 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/table/testLogSplittingAfterMasterRecoveryDueToZKExpiry 2023-07-12 00:32:17,002 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table/testLogSplittingAfterMasterRecoveryDueToZKExpiry 2023-07-12 00:32:17,002 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/table 2023-07-12 00:32:17,002 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table/hbase:namespace 2023-07-12 00:32:17,002 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table/hbase:meta 2023-07-12 00:32:17,002 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 00:32:17,002 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 00:32:17,002 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 00:32:17,002 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached 2023-07-12 00:32:17,002 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot 2023-07-12 00:32:17,002 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 00:32:17,002 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 00:32:17,002 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached 2023-07-12 00:32:17,002 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc 2023-07-12 00:32:17,002 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 00:32:17,002 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 00:32:17,002 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table 2023-07-12 00:32:17,002 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:17,002 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/hbaseid 2023-07-12 00:32:17,003 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace 2023-07-12 00:32:17,003 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/draining 2023-07-12 00:32:17,003 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 00:32:17,003 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/switch 2023-07-12 00:32:17,003 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot 2023-07-12 00:32:17,003 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master-maintenance 2023-07-12 00:32:17,003 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc 2023-07-12 00:32:17,003 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:32:17,003 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL 2023-07-12 00:32:17,003 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:17,005 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 00:32:17,005 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase 2023-07-12 00:32:17,020 INFO [Listener at localhost.localdomain/34481] hbase.ResourceChecker(175): after: TestZooKeeper#testLogSplittingAfterMasterRecoveryDueToZKExpiry Thread=373 (was 323) Potentially hanging thread: RS-EventLoopGroup-16-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-2093218422_17 at /127.0.0.1:56102 [Waiting for operation #23] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-19-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-18-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-2093218422_17 at /127.0.0.1:37128 [Waiting for operation #10] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-15-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-18-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_META-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-2093218422_17 at /127.0.0.1:37226 [Waiting for operation #11] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.8@localhost.localdomain:44295 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-19-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: jenkins-hbase20:33475Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-15-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.6@localhost.localdomain:44295 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.7@localhost.localdomain:44295 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-16-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-15-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-19-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-2093218422_17 at /127.0.0.1:37244 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1879248547) connection to localhost.localdomain/127.0.0.1:44295 from jenkins.hfs.8 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-17-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-17-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-18-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-2093218422_17 at /127.0.0.1:47422 [Waiting for operation #33] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-17-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-16-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase20:0.Chore.1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase20:0.logRoller java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.wal.AbstractWALRoller.run(AbstractWALRoller.java:179) Potentially hanging thread: IPC Client (1879248547) connection to localhost.localdomain/127.0.0.1:44295 from jenkins.hfs.6 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: IPC Client (1879248547) connection to localhost.localdomain/127.0.0.1:44295 from jenkins.hfs.7 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=759 (was 682) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=216 (was 194) - SystemLoadAverage LEAK? -, ProcessCount=173 (was 172) - ProcessCount LEAK? -, AvailableMemoryMB=4879 (was 5389) 2023-07-12 00:32:17,034 INFO [Listener at localhost.localdomain/34481] hbase.ResourceChecker(147): before: TestZooKeeper#testRegionAssignmentAfterMasterRecoveryDueToZKExpiry Thread=373, OpenFileDescriptor=759, MaxFileDescriptor=60000, SystemLoadAverage=216, ProcessCount=173, AvailableMemoryMB=4878 2023-07-12 00:32:17,043 INFO [Listener at localhost.localdomain/34481] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983 with version=8 2023-07-12 00:32:17,043 INFO [Listener at localhost.localdomain/34481] hbase.HBaseTestingUtility(1408): The hbase.fs.tmp.dir is set to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/hbase-staging 2023-07-12 00:32:17,044 INFO [Listener at localhost.localdomain/34481] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-07-12 00:32:17,044 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:17,044 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:17,045 INFO [Listener at localhost.localdomain/34481] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 00:32:17,045 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:17,045 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 00:32:17,045 INFO [Listener at localhost.localdomain/34481] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 00:32:17,045 INFO [Listener at localhost.localdomain/34481] ipc.NettyRpcServer(120): Bind to /148.251.75.209:45037 2023-07-12 00:32:17,046 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:17,046 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:17,047 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=master:45037 connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:32:17,049 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:450370x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:32:17,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:45037-0x1007e974181002a connected 2023-07-12 00:32:17,056 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:17,056 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 00:32:17,057 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 00:32:17,063 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45037 2023-07-12 00:32:17,063 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45037 2023-07-12 00:32:17,064 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45037 2023-07-12 00:32:17,065 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45037 2023-07-12 00:32:17,066 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45037 2023-07-12 00:32:17,067 INFO [Listener at localhost.localdomain/34481] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983, hbase.cluster.distributed=false 2023-07-12 00:32:17,070 INFO [Listener at localhost.localdomain/34481] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-07-12 00:32:17,070 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:17,070 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:17,070 INFO [Listener at localhost.localdomain/34481] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 00:32:17,071 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:17,071 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 00:32:17,071 INFO [Listener at localhost.localdomain/34481] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 00:32:17,073 INFO [Listener at localhost.localdomain/34481] ipc.NettyRpcServer(120): Bind to /148.251.75.209:34407 2023-07-12 00:32:17,074 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:17,075 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:17,076 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=master:34407 connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:32:17,089 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:344070x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:32:17,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:34407-0x1007e974181002b connected 2023-07-12 00:32:17,092 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:17,092 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 00:32:17,092 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 00:32:17,093 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34407 2023-07-12 00:32:17,097 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34407 2023-07-12 00:32:17,098 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34407 2023-07-12 00:32:17,101 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34407 2023-07-12 00:32:17,102 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34407 2023-07-12 00:32:17,102 INFO [Listener at localhost.localdomain/34481] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983, hbase.cluster.distributed=false 2023-07-12 00:32:17,117 INFO [Listener at localhost.localdomain/34481] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-07-12 00:32:17,118 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:17,118 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:17,118 INFO [Listener at localhost.localdomain/34481] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 00:32:17,118 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:17,118 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 00:32:17,118 INFO [Listener at localhost.localdomain/34481] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 00:32:17,119 INFO [Listener at localhost.localdomain/34481] ipc.NettyRpcServer(120): Bind to /148.251.75.209:33511 2023-07-12 00:32:17,119 INFO [Listener at localhost.localdomain/34481] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 00:32:17,121 DEBUG [Listener at localhost.localdomain/34481] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 00:32:17,122 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:17,123 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:17,125 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33511 connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:32:17,135 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:335110x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:32:17,137 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:335110x0, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:17,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:33511-0x1007e974181002c connected 2023-07-12 00:32:17,138 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:33511-0x1007e974181002c, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 00:32:17,139 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:33511-0x1007e974181002c, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 00:32:17,144 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33511 2023-07-12 00:32:17,144 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33511 2023-07-12 00:32:17,145 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33511 2023-07-12 00:32:17,145 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33511 2023-07-12 00:32:17,145 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33511 2023-07-12 00:32:17,155 INFO [Listener at localhost.localdomain/34481] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-07-12 00:32:17,155 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:17,155 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:17,156 INFO [Listener at localhost.localdomain/34481] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 00:32:17,156 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:17,156 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 00:32:17,156 INFO [Listener at localhost.localdomain/34481] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 00:32:17,157 INFO [Listener at localhost.localdomain/34481] ipc.NettyRpcServer(120): Bind to /148.251.75.209:39949 2023-07-12 00:32:17,157 INFO [Listener at localhost.localdomain/34481] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 00:32:17,158 DEBUG [Listener at localhost.localdomain/34481] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 00:32:17,159 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:17,160 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:17,160 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39949 connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:32:17,162 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:399490x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:32:17,164 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:399490x0, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:17,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:39949-0x1007e974181002d connected 2023-07-12 00:32:17,165 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:39949-0x1007e974181002d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 00:32:17,166 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:39949-0x1007e974181002d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 00:32:17,166 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39949 2023-07-12 00:32:17,168 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39949 2023-07-12 00:32:17,170 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39949 2023-07-12 00:32:17,170 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39949 2023-07-12 00:32:17,171 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39949 2023-07-12 00:32:17,176 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,45037,1689121937044 2023-07-12 00:32:17,177 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,34407,1689121937069 2023-07-12 00:32:17,178 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:32:17,178 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:32:17,179 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,34407,1689121937069 2023-07-12 00:32:17,179 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,45037,1689121937044 2023-07-12 00:32:17,180 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:33511-0x1007e974181002c, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:17,180 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:17,180 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:39949-0x1007e974181002d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:17,182 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:17,180 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:17,182 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:17,182 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:32:17,185 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:32:17,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:32:17,186 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,34407,1689121937069 from backup master directory 2023-07-12 00:32:17,186 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(278): Another master is the active master, jenkins-hbase20.apache.org,34407,1689121937069; waiting to become the next active master 2023-07-12 00:32:17,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:32:17,187 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,34407,1689121937069 2023-07-12 00:32:17,187 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:32:17,187 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:32:17,187 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 00:32:17,187 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,34407,1689121937069 2023-07-12 00:32:17,206 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/hbase.id with ID: 74bf7c9a-3df0-4cf0-9276-38fe3d502310 2023-07-12 00:32:17,218 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:17,227 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:17,227 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:17,250 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x4163bcd9 to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:32:17,270 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@257fbc63, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:32:17,270 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 00:32:17,271 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 00:32:17,272 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:32:17,275 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store-tmp 2023-07-12 00:32:17,290 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:17,291 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 00:32:17,291 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:17,291 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:17,291 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 00:32:17,291 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:17,291 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:17,291 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 00:32:17,292 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,34407,1689121937069 2023-07-12 00:32:17,295 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C34407%2C1689121937069, suffix=, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,34407,1689121937069, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/oldWALs, maxLogs=10 2023-07-12 00:32:17,312 DEBUG [RS-EventLoopGroup-23-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:32:17,312 DEBUG [RS-EventLoopGroup-23-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:32:17,314 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,34407,1689121937069/jenkins-hbase20.apache.org%2C34407%2C1689121937069.1689121937295 2023-07-12 00:32:17,322 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK], DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK]] 2023-07-12 00:32:17,323 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 00:32:17,323 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:17,323 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:17,323 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:17,327 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:17,329 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 00:32:17,329 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 00:32:17,330 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:17,331 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:17,331 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:17,334 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 00:32:17,336 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:17,337 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58915507, jitterRate=-0.12209053337574005}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:17,337 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 00:32:17,337 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 00:32:17,339 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-12 00:32:17,339 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-12 00:32:17,339 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 00:32:17,339 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 00:32:17,340 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-07-12 00:32:17,340 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 00:32:17,346 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-12 00:32:17,348 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-12 00:32:17,361 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 00:32:17,362 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 00:32:17,362 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 00:32:17,363 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 00:32:17,364 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:17,364 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:17,365 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 00:32:17,366 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 00:32:17,366 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 00:32:17,367 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 00:32:17,367 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:17,367 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:39949-0x1007e974181002d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 00:32:17,367 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:33511-0x1007e974181002c, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 00:32:17,367 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 00:32:17,367 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:17,371 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,34407,1689121937069, sessionid=0x1007e974181002b, setting cluster-up flag (Was=false) 2023-07-12 00:32:17,380 INFO [M:0;jenkins-hbase20:45037] regionserver.HRegionServer(951): ClusterId : 74bf7c9a-3df0-4cf0-9276-38fe3d502310 2023-07-12 00:32:17,385 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:17,385 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:17,387 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 00:32:17,388 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,34407,1689121937069 2023-07-12 00:32:17,390 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:17,390 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:17,392 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 00:32:17,393 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,34407,1689121937069 2023-07-12 00:32:17,393 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.hbase-snapshot/.tmp 2023-07-12 00:32:17,400 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-12 00:32:17,401 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:32:17,401 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:32:17,401 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:32:17,401 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 00:32:17,401 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-07-12 00:32:17,401 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:17,401 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-12 00:32:17,401 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:17,404 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689121967404 2023-07-12 00:32:17,404 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 00:32:17,404 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 00:32:17,404 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 00:32:17,404 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 00:32:17,404 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 00:32:17,404 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 00:32:17,404 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:17,406 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 00:32:17,406 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 00:32:17,406 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 00:32:17,406 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-12 00:32:17,406 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 00:32:17,406 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 00:32:17,406 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 00:32:17,407 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 00:32:17,410 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689121937406,5,FailOnTimeoutGroup] 2023-07-12 00:32:17,413 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689121937413,5,FailOnTimeoutGroup] 2023-07-12 00:32:17,414 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:17,414 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 00:32:17,414 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:17,414 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:17,425 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 00:32:17,425 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 00:32:17,425 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983 2023-07-12 00:32:17,438 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:17,440 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 00:32:17,441 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/info 2023-07-12 00:32:17,442 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 00:32:17,442 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:17,442 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 00:32:17,444 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/rep_barrier 2023-07-12 00:32:17,444 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 00:32:17,445 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:17,445 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 00:32:17,446 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/table 2023-07-12 00:32:17,446 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 00:32:17,447 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:17,448 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740 2023-07-12 00:32:17,448 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740 2023-07-12 00:32:17,450 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 00:32:17,451 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 00:32:17,454 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:17,454 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72310558, jitterRate=0.07751128077507019}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 00:32:17,454 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 00:32:17,454 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 00:32:17,454 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 00:32:17,455 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 00:32:17,455 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 00:32:17,455 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 00:32:17,458 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 00:32:17,459 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 00:32:17,460 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 00:32:17,460 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-12 00:32:17,460 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-12 00:32:17,462 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-12 00:32:17,463 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-12 00:32:17,476 INFO [RS:1;jenkins-hbase20:39949] regionserver.HRegionServer(951): ClusterId : 74bf7c9a-3df0-4cf0-9276-38fe3d502310 2023-07-12 00:32:17,476 INFO [RS:0;jenkins-hbase20:33511] regionserver.HRegionServer(951): ClusterId : 74bf7c9a-3df0-4cf0-9276-38fe3d502310 2023-07-12 00:32:17,476 DEBUG [RS:1;jenkins-hbase20:39949] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 00:32:17,476 DEBUG [RS:0;jenkins-hbase20:33511] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 00:32:17,478 DEBUG [RS:1;jenkins-hbase20:39949] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 00:32:17,478 DEBUG [RS:1;jenkins-hbase20:39949] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 00:32:17,478 DEBUG [RS:0;jenkins-hbase20:33511] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 00:32:17,478 DEBUG [RS:0;jenkins-hbase20:33511] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 00:32:17,479 DEBUG [RS:1;jenkins-hbase20:39949] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 00:32:17,480 DEBUG [RS:0;jenkins-hbase20:33511] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 00:32:17,482 DEBUG [RS:1;jenkins-hbase20:39949] zookeeper.ReadOnlyZKClient(139): Connect 0x6d055514 to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:32:17,482 DEBUG [RS:0;jenkins-hbase20:33511] zookeeper.ReadOnlyZKClient(139): Connect 0x288369ca to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:32:17,495 DEBUG [RS:1;jenkins-hbase20:39949] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63c2b6c4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:32:17,495 DEBUG [RS:1;jenkins-hbase20:39949] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@67fd3e64, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-12 00:32:17,498 DEBUG [RS:0;jenkins-hbase20:33511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@264492aa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:32:17,498 DEBUG [RS:0;jenkins-hbase20:33511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@227b02ba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-12 00:32:17,506 DEBUG [RS:1;jenkins-hbase20:39949] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase20:39949 2023-07-12 00:32:17,506 INFO [RS:1;jenkins-hbase20:39949] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 00:32:17,506 INFO [RS:1;jenkins-hbase20:39949] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 00:32:17,506 DEBUG [RS:1;jenkins-hbase20:39949] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 00:32:17,507 INFO [RS:1;jenkins-hbase20:39949] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase20.apache.org,34407,1689121937069 with isa=jenkins-hbase20.apache.org/148.251.75.209:39949, startcode=1689121937155 2023-07-12 00:32:17,507 DEBUG [RS:1;jenkins-hbase20:39949] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 00:32:17,508 DEBUG [RS:0;jenkins-hbase20:33511] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:33511 2023-07-12 00:32:17,509 INFO [RS:0;jenkins-hbase20:33511] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 00:32:17,509 INFO [RS:0;jenkins-hbase20:33511] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 00:32:17,509 DEBUG [RS:0;jenkins-hbase20:33511] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 00:32:17,509 INFO [RS:0;jenkins-hbase20:33511] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase20.apache.org,34407,1689121937069 with isa=jenkins-hbase20.apache.org/148.251.75.209:33511, startcode=1689121937117 2023-07-12 00:32:17,509 DEBUG [RS:0;jenkins-hbase20:33511] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 00:32:17,511 INFO [RS-EventLoopGroup-21-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52039, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.10 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 00:32:17,516 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=34407] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,39949,1689121937155 2023-07-12 00:32:17,516 INFO [RS-EventLoopGroup-21-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34821, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.9 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 00:32:17,516 DEBUG [RS:1;jenkins-hbase20:39949] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983 2023-07-12 00:32:17,516 DEBUG [RS:1;jenkins-hbase20:39949] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:44295 2023-07-12 00:32:17,516 DEBUG [RS:1;jenkins-hbase20:39949] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 00:32:17,516 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=34407] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,33511,1689121937117 2023-07-12 00:32:17,517 DEBUG [RS:0;jenkins-hbase20:33511] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983 2023-07-12 00:32:17,517 DEBUG [RS:0;jenkins-hbase20:33511] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:44295 2023-07-12 00:32:17,517 DEBUG [RS:0;jenkins-hbase20:33511] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 00:32:17,517 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:17,518 DEBUG [RS:1;jenkins-hbase20:39949] zookeeper.ZKUtil(162): regionserver:39949-0x1007e974181002d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39949,1689121937155 2023-07-12 00:32:17,518 WARN [RS:1;jenkins-hbase20:39949] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 00:32:17,518 INFO [RS:1;jenkins-hbase20:39949] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:32:17,518 DEBUG [RS:1;jenkins-hbase20:39949] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,39949,1689121937155 2023-07-12 00:32:17,518 DEBUG [RS:0;jenkins-hbase20:33511] zookeeper.ZKUtil(162): regionserver:33511-0x1007e974181002c, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,33511,1689121937117 2023-07-12 00:32:17,518 WARN [RS:0;jenkins-hbase20:33511] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 00:32:17,518 INFO [RS:0;jenkins-hbase20:33511] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:32:17,518 DEBUG [RS:0;jenkins-hbase20:33511] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33511,1689121937117 2023-07-12 00:32:17,520 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,33511,1689121937117] 2023-07-12 00:32:17,520 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,39949,1689121937155] 2023-07-12 00:32:17,536 DEBUG [RS:0;jenkins-hbase20:33511] zookeeper.ZKUtil(162): regionserver:33511-0x1007e974181002c, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,33511,1689121937117 2023-07-12 00:32:17,536 DEBUG [RS:1;jenkins-hbase20:39949] zookeeper.ZKUtil(162): regionserver:39949-0x1007e974181002d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,33511,1689121937117 2023-07-12 00:32:17,537 DEBUG [RS:0;jenkins-hbase20:33511] zookeeper.ZKUtil(162): regionserver:33511-0x1007e974181002c, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39949,1689121937155 2023-07-12 00:32:17,537 DEBUG [RS:1;jenkins-hbase20:39949] zookeeper.ZKUtil(162): regionserver:39949-0x1007e974181002d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39949,1689121937155 2023-07-12 00:32:17,537 DEBUG [RS:0;jenkins-hbase20:33511] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 00:32:17,538 INFO [RS:0;jenkins-hbase20:33511] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 00:32:17,539 DEBUG [RS:1;jenkins-hbase20:39949] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 00:32:17,539 INFO [RS:1;jenkins-hbase20:39949] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 00:32:17,542 INFO [RS:0;jenkins-hbase20:33511] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 00:32:17,544 INFO [RS:1;jenkins-hbase20:39949] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 00:32:17,544 INFO [RS:0;jenkins-hbase20:33511] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 00:32:17,544 INFO [RS:0;jenkins-hbase20:33511] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:17,544 INFO [RS:1;jenkins-hbase20:39949] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 00:32:17,544 INFO [RS:1;jenkins-hbase20:39949] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:17,546 INFO [RS:0;jenkins-hbase20:33511] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 00:32:17,549 INFO [RS:1;jenkins-hbase20:39949] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 00:32:17,551 INFO [RS:0;jenkins-hbase20:33511] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:17,551 INFO [RS:1;jenkins-hbase20:39949] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:17,552 DEBUG [RS:0;jenkins-hbase20:33511] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:17,552 DEBUG [RS:1;jenkins-hbase20:39949] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:17,552 DEBUG [RS:0;jenkins-hbase20:33511] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:17,552 DEBUG [RS:1;jenkins-hbase20:39949] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:17,552 DEBUG [RS:0;jenkins-hbase20:33511] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:17,552 DEBUG [RS:1;jenkins-hbase20:39949] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:17,552 DEBUG [RS:0;jenkins-hbase20:33511] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:17,553 DEBUG [RS:1;jenkins-hbase20:39949] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:17,553 DEBUG [RS:0;jenkins-hbase20:33511] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:17,553 DEBUG [RS:1;jenkins-hbase20:39949] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:17,553 DEBUG [RS:0;jenkins-hbase20:33511] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-12 00:32:17,553 DEBUG [RS:1;jenkins-hbase20:39949] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-12 00:32:17,553 DEBUG [RS:0;jenkins-hbase20:33511] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:17,553 DEBUG [RS:1;jenkins-hbase20:39949] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:17,553 DEBUG [RS:0;jenkins-hbase20:33511] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:17,553 DEBUG [RS:1;jenkins-hbase20:39949] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:17,553 DEBUG [RS:0;jenkins-hbase20:33511] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:17,553 DEBUG [RS:1;jenkins-hbase20:39949] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:17,553 DEBUG [RS:0;jenkins-hbase20:33511] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:17,553 DEBUG [RS:1;jenkins-hbase20:39949] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:17,556 INFO [RS:0;jenkins-hbase20:33511] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:17,556 INFO [RS:0;jenkins-hbase20:33511] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:17,556 INFO [RS:0;jenkins-hbase20:33511] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:17,562 INFO [RS:1;jenkins-hbase20:39949] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:17,563 INFO [RS:1;jenkins-hbase20:39949] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:17,563 INFO [RS:1;jenkins-hbase20:39949] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:17,568 INFO [RS:0;jenkins-hbase20:33511] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 00:32:17,568 INFO [RS:0;jenkins-hbase20:33511] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,33511,1689121937117-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:17,575 INFO [RS:1;jenkins-hbase20:39949] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 00:32:17,576 INFO [RS:1;jenkins-hbase20:39949] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,39949,1689121937155-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:17,579 INFO [RS:0;jenkins-hbase20:33511] regionserver.Replication(203): jenkins-hbase20.apache.org,33511,1689121937117 started 2023-07-12 00:32:17,579 INFO [RS:0;jenkins-hbase20:33511] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,33511,1689121937117, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:33511, sessionid=0x1007e974181002c 2023-07-12 00:32:17,579 DEBUG [RS:0;jenkins-hbase20:33511] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 00:32:17,579 DEBUG [RS:0;jenkins-hbase20:33511] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,33511,1689121937117 2023-07-12 00:32:17,579 DEBUG [RS:0;jenkins-hbase20:33511] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,33511,1689121937117' 2023-07-12 00:32:17,580 DEBUG [RS:0;jenkins-hbase20:33511] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 00:32:17,580 DEBUG [RS:0;jenkins-hbase20:33511] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 00:32:17,580 DEBUG [RS:0;jenkins-hbase20:33511] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 00:32:17,580 DEBUG [RS:0;jenkins-hbase20:33511] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 00:32:17,580 DEBUG [RS:0;jenkins-hbase20:33511] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,33511,1689121937117 2023-07-12 00:32:17,580 DEBUG [RS:0;jenkins-hbase20:33511] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,33511,1689121937117' 2023-07-12 00:32:17,580 DEBUG [RS:0;jenkins-hbase20:33511] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 00:32:17,581 DEBUG [RS:0;jenkins-hbase20:33511] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 00:32:17,581 DEBUG [RS:0;jenkins-hbase20:33511] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 00:32:17,581 INFO [RS:0;jenkins-hbase20:33511] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 00:32:17,581 INFO [RS:0;jenkins-hbase20:33511] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 00:32:17,585 INFO [RS:1;jenkins-hbase20:39949] regionserver.Replication(203): jenkins-hbase20.apache.org,39949,1689121937155 started 2023-07-12 00:32:17,585 INFO [RS:1;jenkins-hbase20:39949] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,39949,1689121937155, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:39949, sessionid=0x1007e974181002d 2023-07-12 00:32:17,585 DEBUG [RS:1;jenkins-hbase20:39949] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 00:32:17,585 DEBUG [RS:1;jenkins-hbase20:39949] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,39949,1689121937155 2023-07-12 00:32:17,585 DEBUG [RS:1;jenkins-hbase20:39949] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,39949,1689121937155' 2023-07-12 00:32:17,585 DEBUG [RS:1;jenkins-hbase20:39949] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 00:32:17,585 DEBUG [RS:1;jenkins-hbase20:39949] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 00:32:17,586 DEBUG [RS:1;jenkins-hbase20:39949] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 00:32:17,586 DEBUG [RS:1;jenkins-hbase20:39949] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 00:32:17,586 DEBUG [RS:1;jenkins-hbase20:39949] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,39949,1689121937155 2023-07-12 00:32:17,586 DEBUG [RS:1;jenkins-hbase20:39949] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,39949,1689121937155' 2023-07-12 00:32:17,586 DEBUG [RS:1;jenkins-hbase20:39949] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 00:32:17,586 DEBUG [RS:1;jenkins-hbase20:39949] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 00:32:17,586 DEBUG [RS:1;jenkins-hbase20:39949] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 00:32:17,586 INFO [RS:1;jenkins-hbase20:39949] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 00:32:17,586 INFO [RS:1;jenkins-hbase20:39949] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 00:32:17,613 DEBUG [jenkins-hbase20:34407] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-12 00:32:17,614 DEBUG [jenkins-hbase20:34407] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-07-12 00:32:17,615 DEBUG [jenkins-hbase20:34407] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 00:32:17,615 DEBUG [jenkins-hbase20:34407] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 00:32:17,615 DEBUG [jenkins-hbase20:34407] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 00:32:17,616 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,39949,1689121937155, state=OPENING 2023-07-12 00:32:17,617 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-12 00:32:17,618 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:17,618 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:17,618 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 00:32:17,618 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 00:32:17,618 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,39949,1689121937155}] 2023-07-12 00:32:17,685 INFO [RS:0;jenkins-hbase20:33511] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C33511%2C1689121937117, suffix=, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33511,1689121937117, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/oldWALs, maxLogs=32 2023-07-12 00:32:17,690 INFO [RS:1;jenkins-hbase20:39949] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C39949%2C1689121937155, suffix=, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,39949,1689121937155, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/oldWALs, maxLogs=32 2023-07-12 00:32:17,705 DEBUG [RS-EventLoopGroup-23-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:32:17,706 DEBUG [RS-EventLoopGroup-23-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:32:17,708 DEBUG [RS-EventLoopGroup-23-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:32:17,708 DEBUG [RS-EventLoopGroup-23-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:32:17,708 INFO [RS:0;jenkins-hbase20:33511] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,33511,1689121937117/jenkins-hbase20.apache.org%2C33511%2C1689121937117.1689121937686 2023-07-12 00:32:17,709 DEBUG [RS:0;jenkins-hbase20:33511] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK], DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK]] 2023-07-12 00:32:17,710 INFO [RS:1;jenkins-hbase20:39949] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,39949,1689121937155/jenkins-hbase20.apache.org%2C39949%2C1689121937155.1689121937691 2023-07-12 00:32:17,711 DEBUG [RS:1;jenkins-hbase20:39949] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK], DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK]] 2023-07-12 00:32:17,773 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,39949,1689121937155 2023-07-12 00:32:17,773 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 00:32:17,776 INFO [RS-EventLoopGroup-23-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44204, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 00:32:17,783 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-12 00:32:17,783 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:32:17,786 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C39949%2C1689121937155.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,39949,1689121937155, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/oldWALs, maxLogs=32 2023-07-12 00:32:17,797 DEBUG [RS-EventLoopGroup-23-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:32:17,797 DEBUG [RS-EventLoopGroup-23-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:32:17,799 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,39949,1689121937155/jenkins-hbase20.apache.org%2C39949%2C1689121937155.meta.1689121937786.meta 2023-07-12 00:32:17,799 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK], DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK]] 2023-07-12 00:32:17,799 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-12 00:32:17,799 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-12 00:32:17,799 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-12 00:32:17,799 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-12 00:32:17,799 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-12 00:32:17,800 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:17,800 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-12 00:32:17,800 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-12 00:32:17,801 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 00:32:17,802 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/info 2023-07-12 00:32:17,802 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/info 2023-07-12 00:32:17,802 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 00:32:17,803 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:17,803 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 00:32:17,804 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/rep_barrier 2023-07-12 00:32:17,804 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/rep_barrier 2023-07-12 00:32:17,804 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 00:32:17,805 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:17,805 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 00:32:17,806 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/table 2023-07-12 00:32:17,806 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740/table 2023-07-12 00:32:17,806 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 00:32:17,806 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:17,807 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740 2023-07-12 00:32:17,808 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/meta/1588230740 2023-07-12 00:32:17,811 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 00:32:17,812 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 00:32:17,813 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61882573, jitterRate=-0.07787780463695526}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 00:32:17,813 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 00:32:17,814 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689121937773 2023-07-12 00:32:17,818 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-12 00:32:17,819 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-12 00:32:17,819 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,39949,1689121937155, state=OPEN 2023-07-12 00:32:17,820 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 00:32:17,820 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 00:32:17,820 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 00:32:17,821 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 00:32:17,823 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-12 00:32:17,823 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,39949,1689121937155 in 202 msec 2023-07-12 00:32:17,825 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-12 00:32:17,825 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 363 msec 2023-07-12 00:32:17,827 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 427 msec 2023-07-12 00:32:17,827 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689121937827, completionTime=-1 2023-07-12 00:32:17,827 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-12 00:32:17,827 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 00:32:17,833 DEBUG [hconnection-0x6563e7ff-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 00:32:17,834 INFO [RS-EventLoopGroup-23-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44216, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 00:32:17,835 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-12 00:32:17,835 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689121997835 2023-07-12 00:32:17,835 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689122057835 2023-07-12 00:32:17,836 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 8 msec 2023-07-12 00:32:17,841 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34407,1689121937069-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:17,841 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34407,1689121937069-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:17,841 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34407,1689121937069-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:17,841 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:34407, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:17,841 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:17,841 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-12 00:32:17,841 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 00:32:17,842 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-12 00:32:17,844 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 00:32:17,847 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 00:32:17,848 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 00:32:17,850 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/hbase/namespace/ed9da68fb58a11fa382831aaae424ddd 2023-07-12 00:32:17,851 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/hbase/namespace/ed9da68fb58a11fa382831aaae424ddd empty. 2023-07-12 00:32:17,852 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/hbase/namespace/ed9da68fb58a11fa382831aaae424ddd 2023-07-12 00:32:17,852 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-12 00:32:17,867 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-12 00:32:17,868 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => ed9da68fb58a11fa382831aaae424ddd, NAME => 'hbase:namespace,,1689121937841.ed9da68fb58a11fa382831aaae424ddd.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp 2023-07-12 00:32:17,875 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689121937841.ed9da68fb58a11fa382831aaae424ddd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:17,875 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing ed9da68fb58a11fa382831aaae424ddd, disabling compactions & flushes 2023-07-12 00:32:17,875 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689121937841.ed9da68fb58a11fa382831aaae424ddd. 2023-07-12 00:32:17,876 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689121937841.ed9da68fb58a11fa382831aaae424ddd. 2023-07-12 00:32:17,876 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689121937841.ed9da68fb58a11fa382831aaae424ddd. after waiting 0 ms 2023-07-12 00:32:17,876 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689121937841.ed9da68fb58a11fa382831aaae424ddd. 2023-07-12 00:32:17,876 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689121937841.ed9da68fb58a11fa382831aaae424ddd. 2023-07-12 00:32:17,876 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for ed9da68fb58a11fa382831aaae424ddd: 2023-07-12 00:32:17,878 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 00:32:17,879 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689121937841.ed9da68fb58a11fa382831aaae424ddd.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689121937879"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689121937879"}]},"ts":"1689121937879"} 2023-07-12 00:32:17,881 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 00:32:17,882 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 00:32:17,882 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689121937882"}]},"ts":"1689121937882"} 2023-07-12 00:32:17,884 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-12 00:32:17,886 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-07-12 00:32:17,887 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 00:32:17,887 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 00:32:17,887 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 00:32:17,887 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=ed9da68fb58a11fa382831aaae424ddd, ASSIGN}] 2023-07-12 00:32:17,890 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=ed9da68fb58a11fa382831aaae424ddd, ASSIGN 2023-07-12 00:32:17,891 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=ed9da68fb58a11fa382831aaae424ddd, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,39949,1689121937155; forceNewPlan=false, retain=false 2023-07-12 00:32:18,041 INFO [jenkins-hbase20:34407] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-12 00:32:18,042 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=ed9da68fb58a11fa382831aaae424ddd, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39949,1689121937155 2023-07-12 00:32:18,042 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689121937841.ed9da68fb58a11fa382831aaae424ddd.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689121938042"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121938042"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121938042"}]},"ts":"1689121938042"} 2023-07-12 00:32:18,044 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure ed9da68fb58a11fa382831aaae424ddd, server=jenkins-hbase20.apache.org,39949,1689121937155}] 2023-07-12 00:32:18,200 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689121937841.ed9da68fb58a11fa382831aaae424ddd. 2023-07-12 00:32:18,200 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ed9da68fb58a11fa382831aaae424ddd, NAME => 'hbase:namespace,,1689121937841.ed9da68fb58a11fa382831aaae424ddd.', STARTKEY => '', ENDKEY => ''} 2023-07-12 00:32:18,200 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace ed9da68fb58a11fa382831aaae424ddd 2023-07-12 00:32:18,200 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689121937841.ed9da68fb58a11fa382831aaae424ddd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:18,201 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for ed9da68fb58a11fa382831aaae424ddd 2023-07-12 00:32:18,201 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for ed9da68fb58a11fa382831aaae424ddd 2023-07-12 00:32:18,202 INFO [StoreOpener-ed9da68fb58a11fa382831aaae424ddd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region ed9da68fb58a11fa382831aaae424ddd 2023-07-12 00:32:18,203 DEBUG [StoreOpener-ed9da68fb58a11fa382831aaae424ddd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/ed9da68fb58a11fa382831aaae424ddd/info 2023-07-12 00:32:18,203 DEBUG [StoreOpener-ed9da68fb58a11fa382831aaae424ddd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/ed9da68fb58a11fa382831aaae424ddd/info 2023-07-12 00:32:18,204 INFO [StoreOpener-ed9da68fb58a11fa382831aaae424ddd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ed9da68fb58a11fa382831aaae424ddd columnFamilyName info 2023-07-12 00:32:18,204 INFO [StoreOpener-ed9da68fb58a11fa382831aaae424ddd-1] regionserver.HStore(310): Store=ed9da68fb58a11fa382831aaae424ddd/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:18,205 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/ed9da68fb58a11fa382831aaae424ddd 2023-07-12 00:32:18,205 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/ed9da68fb58a11fa382831aaae424ddd 2023-07-12 00:32:18,207 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for ed9da68fb58a11fa382831aaae424ddd 2023-07-12 00:32:18,209 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/hbase/namespace/ed9da68fb58a11fa382831aaae424ddd/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:18,210 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened ed9da68fb58a11fa382831aaae424ddd; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61195969, jitterRate=-0.08810900151729584}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:18,210 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for ed9da68fb58a11fa382831aaae424ddd: 2023-07-12 00:32:18,210 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689121937841.ed9da68fb58a11fa382831aaae424ddd., pid=6, masterSystemTime=1689121938196 2023-07-12 00:32:18,212 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689121937841.ed9da68fb58a11fa382831aaae424ddd. 2023-07-12 00:32:18,212 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689121937841.ed9da68fb58a11fa382831aaae424ddd. 2023-07-12 00:32:18,212 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=ed9da68fb58a11fa382831aaae424ddd, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,39949,1689121937155 2023-07-12 00:32:18,213 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689121937841.ed9da68fb58a11fa382831aaae424ddd.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689121938212"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121938212"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121938212"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121938212"}]},"ts":"1689121938212"} 2023-07-12 00:32:18,216 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-12 00:32:18,216 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure ed9da68fb58a11fa382831aaae424ddd, server=jenkins-hbase20.apache.org,39949,1689121937155 in 170 msec 2023-07-12 00:32:18,218 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-12 00:32:18,218 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=ed9da68fb58a11fa382831aaae424ddd, ASSIGN in 329 msec 2023-07-12 00:32:18,218 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 00:32:18,219 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689121938218"}]},"ts":"1689121938218"} 2023-07-12 00:32:18,220 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-12 00:32:18,222 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 00:32:18,223 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 381 msec 2023-07-12 00:32:18,244 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-12 00:32:18,247 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:18,247 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-12 00:32:18,247 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:18,250 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-12 00:32:18,269 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 00:32:18,274 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 23 msec 2023-07-12 00:32:18,283 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-12 00:32:18,292 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 00:32:18,296 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 13 msec 2023-07-12 00:32:18,308 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 00:32:18,309 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 00:32:18,309 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 1.121sec 2023-07-12 00:32:18,310 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 00:32:18,310 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 00:32:18,310 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 00:32:18,311 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34407,1689121937069-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 00:32:18,311 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34407,1689121937069-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 00:32:18,315 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 00:32:18,377 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ReadOnlyZKClient(139): Connect 0x3f696882 to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:32:18,381 DEBUG [Listener at localhost.localdomain/34481] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ed578e9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:32:18,382 DEBUG [hconnection-0xce4af5a-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 00:32:18,384 INFO [RS-EventLoopGroup-23-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44218, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 00:32:18,386 INFO [Listener at localhost.localdomain/34481] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,34407,1689121937069 2023-07-12 00:32:18,396 INFO [Listener at localhost.localdomain/34481] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-07-12 00:32:18,396 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:18,396 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:18,396 INFO [Listener at localhost.localdomain/34481] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 00:32:18,397 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 00:32:18,397 INFO [Listener at localhost.localdomain/34481] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 00:32:18,397 INFO [Listener at localhost.localdomain/34481] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 00:32:18,397 INFO [Listener at localhost.localdomain/34481] ipc.NettyRpcServer(120): Bind to /148.251.75.209:41331 2023-07-12 00:32:18,398 INFO [Listener at localhost.localdomain/34481] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 00:32:18,400 DEBUG [Listener at localhost.localdomain/34481] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 00:32:18,400 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:18,401 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:18,402 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:41331 connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:32:18,404 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:413310x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:32:18,408 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): regionserver:413310x0, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:32:18,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:41331-0x1007e9741810032 connected 2023-07-12 00:32:18,409 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): regionserver:41331-0x1007e9741810032, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-12 00:32:18,410 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(164): regionserver:41331-0x1007e9741810032, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 00:32:18,411 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41331 2023-07-12 00:32:18,411 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41331 2023-07-12 00:32:18,411 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41331 2023-07-12 00:32:18,412 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41331 2023-07-12 00:32:18,412 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41331 2023-07-12 00:32:18,414 INFO [RS:2;jenkins-hbase20:41331] regionserver.HRegionServer(951): ClusterId : 74bf7c9a-3df0-4cf0-9276-38fe3d502310 2023-07-12 00:32:18,414 DEBUG [RS:2;jenkins-hbase20:41331] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 00:32:18,423 DEBUG [RS:2;jenkins-hbase20:41331] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 00:32:18,423 DEBUG [RS:2;jenkins-hbase20:41331] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 00:32:18,427 DEBUG [RS:2;jenkins-hbase20:41331] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 00:32:18,429 DEBUG [RS:2;jenkins-hbase20:41331] zookeeper.ReadOnlyZKClient(139): Connect 0x15ad5174 to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:32:18,432 DEBUG [RS:2;jenkins-hbase20:41331] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1b6aadb1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:32:18,432 DEBUG [RS:2;jenkins-hbase20:41331] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f553a04, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-12 00:32:18,439 DEBUG [RS:2;jenkins-hbase20:41331] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase20:41331 2023-07-12 00:32:18,439 INFO [RS:2;jenkins-hbase20:41331] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 00:32:18,439 INFO [RS:2;jenkins-hbase20:41331] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 00:32:18,439 DEBUG [RS:2;jenkins-hbase20:41331] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 00:32:18,440 INFO [RS:2;jenkins-hbase20:41331] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase20.apache.org,34407,1689121937069 with isa=jenkins-hbase20.apache.org/148.251.75.209:41331, startcode=1689121938396 2023-07-12 00:32:18,440 DEBUG [RS:2;jenkins-hbase20:41331] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 00:32:18,441 INFO [RS-EventLoopGroup-21-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34077, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.11 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 00:32:18,442 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=34407] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,41331,1689121938396 2023-07-12 00:32:18,442 DEBUG [RS:2;jenkins-hbase20:41331] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983 2023-07-12 00:32:18,442 DEBUG [RS:2;jenkins-hbase20:41331] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:44295 2023-07-12 00:32:18,442 DEBUG [RS:2;jenkins-hbase20:41331] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 00:32:18,444 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:33511-0x1007e974181002c, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:18,444 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:18,444 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:39949-0x1007e974181002d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:18,444 DEBUG [RS:2;jenkins-hbase20:41331] zookeeper.ZKUtil(162): regionserver:41331-0x1007e9741810032, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,41331,1689121938396 2023-07-12 00:32:18,444 WARN [RS:2;jenkins-hbase20:41331] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 00:32:18,444 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,41331,1689121938396] 2023-07-12 00:32:18,444 INFO [RS:2;jenkins-hbase20:41331] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:32:18,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33511-0x1007e974181002c, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,33511,1689121937117 2023-07-12 00:32:18,444 DEBUG [RS:2;jenkins-hbase20:41331] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,41331,1689121938396 2023-07-12 00:32:18,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39949-0x1007e974181002d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,33511,1689121937117 2023-07-12 00:32:18,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33511-0x1007e974181002c, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,41331,1689121938396 2023-07-12 00:32:18,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39949-0x1007e974181002d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,41331,1689121938396 2023-07-12 00:32:18,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33511-0x1007e974181002c, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39949,1689121937155 2023-07-12 00:32:18,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39949-0x1007e974181002d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39949,1689121937155 2023-07-12 00:32:18,449 DEBUG [RS:2;jenkins-hbase20:41331] zookeeper.ZKUtil(162): regionserver:41331-0x1007e9741810032, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,33511,1689121937117 2023-07-12 00:32:18,449 DEBUG [RS:2;jenkins-hbase20:41331] zookeeper.ZKUtil(162): regionserver:41331-0x1007e9741810032, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,41331,1689121938396 2023-07-12 00:32:18,449 DEBUG [RS:2;jenkins-hbase20:41331] zookeeper.ZKUtil(162): regionserver:41331-0x1007e9741810032, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39949,1689121937155 2023-07-12 00:32:18,450 DEBUG [RS:2;jenkins-hbase20:41331] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 00:32:18,450 INFO [RS:2;jenkins-hbase20:41331] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 00:32:18,451 INFO [RS:2;jenkins-hbase20:41331] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 00:32:18,453 INFO [RS:2;jenkins-hbase20:41331] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 00:32:18,453 INFO [RS:2;jenkins-hbase20:41331] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:18,453 INFO [RS:2;jenkins-hbase20:41331] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 00:32:18,457 INFO [RS:2;jenkins-hbase20:41331] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:18,458 DEBUG [RS:2;jenkins-hbase20:41331] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:18,458 DEBUG [RS:2;jenkins-hbase20:41331] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:18,458 DEBUG [RS:2;jenkins-hbase20:41331] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:18,458 DEBUG [RS:2;jenkins-hbase20:41331] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:18,458 DEBUG [RS:2;jenkins-hbase20:41331] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:18,458 DEBUG [RS:2;jenkins-hbase20:41331] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-12 00:32:18,458 DEBUG [RS:2;jenkins-hbase20:41331] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:18,458 DEBUG [RS:2;jenkins-hbase20:41331] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:18,458 DEBUG [RS:2;jenkins-hbase20:41331] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:18,458 DEBUG [RS:2;jenkins-hbase20:41331] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 00:32:18,460 INFO [RS:2;jenkins-hbase20:41331] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:18,460 INFO [RS:2;jenkins-hbase20:41331] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:18,460 INFO [RS:2;jenkins-hbase20:41331] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:18,470 INFO [RS:2;jenkins-hbase20:41331] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 00:32:18,470 INFO [RS:2;jenkins-hbase20:41331] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,41331,1689121938396-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 00:32:18,481 INFO [RS:2;jenkins-hbase20:41331] regionserver.Replication(203): jenkins-hbase20.apache.org,41331,1689121938396 started 2023-07-12 00:32:18,481 INFO [RS:2;jenkins-hbase20:41331] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,41331,1689121938396, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:41331, sessionid=0x1007e9741810032 2023-07-12 00:32:18,481 DEBUG [RS:2;jenkins-hbase20:41331] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 00:32:18,481 DEBUG [RS:2;jenkins-hbase20:41331] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,41331,1689121938396 2023-07-12 00:32:18,481 DEBUG [RS:2;jenkins-hbase20:41331] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,41331,1689121938396' 2023-07-12 00:32:18,481 DEBUG [RS:2;jenkins-hbase20:41331] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 00:32:18,482 DEBUG [RS:2;jenkins-hbase20:41331] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 00:32:18,484 DEBUG [RS:2;jenkins-hbase20:41331] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 00:32:18,484 DEBUG [RS:2;jenkins-hbase20:41331] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 00:32:18,484 DEBUG [RS:2;jenkins-hbase20:41331] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,41331,1689121938396 2023-07-12 00:32:18,484 DEBUG [RS:2;jenkins-hbase20:41331] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,41331,1689121938396' 2023-07-12 00:32:18,484 DEBUG [RS:2;jenkins-hbase20:41331] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 00:32:18,484 DEBUG [Listener at localhost.localdomain/34481] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-12 00:32:18,484 DEBUG [RS:2;jenkins-hbase20:41331] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 00:32:18,485 DEBUG [RS:2;jenkins-hbase20:41331] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 00:32:18,485 INFO [RS:2;jenkins-hbase20:41331] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 00:32:18,485 INFO [RS:2;jenkins-hbase20:41331] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 00:32:18,486 INFO [RS-EventLoopGroup-21-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60750, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-12 00:32:18,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34407] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 00:32:18,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34407] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2023-07-12 00:32:18,492 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 00:32:18,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34407] master.MasterRpcServices(700): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "testRegionAssignmentAfterMasterRecoveryDueToZKExpiry" procId is: 9 2023-07-12 00:32:18,493 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 00:32:18,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34407] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 00:32:18,505 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/7a3cc4296899653225169eafd8b2a036 2023-07-12 00:32:18,505 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/700bb8bb9688a27326bcb98b7132a8d7 2023-07-12 00:32:18,505 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/4bf803cbe0da319f9582ad75165052ed 2023-07-12 00:32:18,505 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/591a2fafd71a4502f0c9d7c65c6de8cd 2023-07-12 00:32:18,505 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/b36a989038618469dbf3c92ec0b243a1 2023-07-12 00:32:18,505 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/b4ca85da7c922b089aeee8eea982fedf 2023-07-12 00:32:18,505 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/02bb0fac148ca555b93632ec43604318 2023-07-12 00:32:18,505 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/491b73821b1517947c4dd95c17aad62c 2023-07-12 00:32:18,506 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/7a3cc4296899653225169eafd8b2a036 empty. 2023-07-12 00:32:18,506 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/b36a989038618469dbf3c92ec0b243a1 empty. 2023-07-12 00:32:18,506 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/02bb0fac148ca555b93632ec43604318 empty. 2023-07-12 00:32:18,506 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/591a2fafd71a4502f0c9d7c65c6de8cd empty. 2023-07-12 00:32:18,506 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/4bf803cbe0da319f9582ad75165052ed empty. 2023-07-12 00:32:18,506 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/700bb8bb9688a27326bcb98b7132a8d7 empty. 2023-07-12 00:32:18,506 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/b36a989038618469dbf3c92ec0b243a1 2023-07-12 00:32:18,506 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/591a2fafd71a4502f0c9d7c65c6de8cd 2023-07-12 00:32:18,506 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/6b5039c25691704ddb23db278cb8e6c5 2023-07-12 00:32:18,506 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/d91c406970e2b1461f63e6cb82f78d8d 2023-07-12 00:32:18,507 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/491b73821b1517947c4dd95c17aad62c empty. 2023-07-12 00:32:18,507 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/b4ca85da7c922b089aeee8eea982fedf empty. 2023-07-12 00:32:18,507 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/7a3cc4296899653225169eafd8b2a036 2023-07-12 00:32:18,507 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/700bb8bb9688a27326bcb98b7132a8d7 2023-07-12 00:32:18,507 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/4bf803cbe0da319f9582ad75165052ed 2023-07-12 00:32:18,507 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/335cd564cec52505cf0fa421b43d2650 2023-07-12 00:32:18,507 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/02bb0fac148ca555b93632ec43604318 2023-07-12 00:32:18,508 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/b4ca85da7c922b089aeee8eea982fedf 2023-07-12 00:32:18,508 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/335cd564cec52505cf0fa421b43d2650 empty. 2023-07-12 00:32:18,508 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/6b5039c25691704ddb23db278cb8e6c5 empty. 2023-07-12 00:32:18,508 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/491b73821b1517947c4dd95c17aad62c 2023-07-12 00:32:18,508 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/d91c406970e2b1461f63e6cb82f78d8d empty. 2023-07-12 00:32:18,508 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/335cd564cec52505cf0fa421b43d2650 2023-07-12 00:32:18,508 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/6b5039c25691704ddb23db278cb8e6c5 2023-07-12 00:32:18,508 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/d91c406970e2b1461f63e6cb82f78d8d 2023-07-12 00:32:18,508 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testRegionAssignmentAfterMasterRecoveryDueToZKExpiry regions 2023-07-12 00:32:18,518 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/.tabledesc/.tableinfo.0000000001 2023-07-12 00:32:18,520 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(7675): creating {ENCODED => 491b73821b1517947c4dd95c17aad62c, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1689121938488.491b73821b1517947c4dd95c17aad62c.', STARTKEY => 'a', ENDKEY => 'b'}, tableDescriptor='testRegionAssignmentAfterMasterRecoveryDueToZKExpiry', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp 2023-07-12 00:32:18,520 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(7675): creating {ENCODED => 02bb0fac148ca555b93632ec43604318, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1689121938488.02bb0fac148ca555b93632ec43604318.', STARTKEY => 'b', ENDKEY => 'c'}, tableDescriptor='testRegionAssignmentAfterMasterRecoveryDueToZKExpiry', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp 2023-07-12 00:32:18,520 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(7675): creating {ENCODED => 7a3cc4296899653225169eafd8b2a036, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1689121938488.7a3cc4296899653225169eafd8b2a036.', STARTKEY => '', ENDKEY => 'a'}, tableDescriptor='testRegionAssignmentAfterMasterRecoveryDueToZKExpiry', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp 2023-07-12 00:32:18,529 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(866): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1689121938488.7a3cc4296899653225169eafd8b2a036.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:18,529 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1604): Closing 7a3cc4296899653225169eafd8b2a036, disabling compactions & flushes 2023-07-12 00:32:18,529 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1626): Closing region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1689121938488.7a3cc4296899653225169eafd8b2a036. 2023-07-12 00:32:18,529 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1689121938488.7a3cc4296899653225169eafd8b2a036. 2023-07-12 00:32:18,529 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1714): Acquired close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1689121938488.7a3cc4296899653225169eafd8b2a036. after waiting 0 ms 2023-07-12 00:32:18,529 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1724): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1689121938488.7a3cc4296899653225169eafd8b2a036. 2023-07-12 00:32:18,529 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1838): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1689121938488.7a3cc4296899653225169eafd8b2a036. 2023-07-12 00:32:18,529 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1558): Region close journal for 7a3cc4296899653225169eafd8b2a036: 2023-07-12 00:32:18,530 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(7675): creating {ENCODED => b4ca85da7c922b089aeee8eea982fedf, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1689121938488.b4ca85da7c922b089aeee8eea982fedf.', STARTKEY => 'c', ENDKEY => 'd'}, tableDescriptor='testRegionAssignmentAfterMasterRecoveryDueToZKExpiry', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp 2023-07-12 00:32:18,531 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(866): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1689121938488.491b73821b1517947c4dd95c17aad62c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:18,531 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(866): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1689121938488.02bb0fac148ca555b93632ec43604318.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:18,531 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1604): Closing 491b73821b1517947c4dd95c17aad62c, disabling compactions & flushes 2023-07-12 00:32:18,531 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1604): Closing 02bb0fac148ca555b93632ec43604318, disabling compactions & flushes 2023-07-12 00:32:18,531 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1626): Closing region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1689121938488.491b73821b1517947c4dd95c17aad62c. 2023-07-12 00:32:18,531 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1626): Closing region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1689121938488.02bb0fac148ca555b93632ec43604318. 2023-07-12 00:32:18,531 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1689121938488.02bb0fac148ca555b93632ec43604318. 2023-07-12 00:32:18,531 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1689121938488.491b73821b1517947c4dd95c17aad62c. 2023-07-12 00:32:18,532 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1714): Acquired close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1689121938488.02bb0fac148ca555b93632ec43604318. after waiting 0 ms 2023-07-12 00:32:18,532 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1724): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1689121938488.02bb0fac148ca555b93632ec43604318. 2023-07-12 00:32:18,532 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1714): Acquired close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1689121938488.491b73821b1517947c4dd95c17aad62c. after waiting 0 ms 2023-07-12 00:32:18,532 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1838): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1689121938488.02bb0fac148ca555b93632ec43604318. 2023-07-12 00:32:18,532 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1558): Region close journal for 02bb0fac148ca555b93632ec43604318: 2023-07-12 00:32:18,532 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1724): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1689121938488.491b73821b1517947c4dd95c17aad62c. 2023-07-12 00:32:18,532 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1838): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1689121938488.491b73821b1517947c4dd95c17aad62c. 2023-07-12 00:32:18,532 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1558): Region close journal for 491b73821b1517947c4dd95c17aad62c: 2023-07-12 00:32:18,532 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(7675): creating {ENCODED => b36a989038618469dbf3c92ec0b243a1, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1689121938488.b36a989038618469dbf3c92ec0b243a1.', STARTKEY => 'd', ENDKEY => 'e'}, tableDescriptor='testRegionAssignmentAfterMasterRecoveryDueToZKExpiry', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp 2023-07-12 00:32:18,532 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(7675): creating {ENCODED => 700bb8bb9688a27326bcb98b7132a8d7, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1689121938488.700bb8bb9688a27326bcb98b7132a8d7.', STARTKEY => 'e', ENDKEY => 'f'}, tableDescriptor='testRegionAssignmentAfterMasterRecoveryDueToZKExpiry', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp 2023-07-12 00:32:18,587 INFO [RS:2;jenkins-hbase20:41331] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C41331%2C1689121938396, suffix=, logDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,41331,1689121938396, archiveDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/oldWALs, maxLogs=32 2023-07-12 00:32:18,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34407] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 00:32:18,605 DEBUG [RS-EventLoopGroup-24-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK] 2023-07-12 00:32:18,605 DEBUG [RS-EventLoopGroup-24-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK] 2023-07-12 00:32:18,607 INFO [RS:2;jenkins-hbase20:41331] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,41331,1689121938396/jenkins-hbase20.apache.org%2C41331%2C1689121938396.1689121938587 2023-07-12 00:32:18,607 DEBUG [RS:2;jenkins-hbase20:41331] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41473,DS-24609d7a-0366-47b4-b383-81fc18b4b2eb,DISK], DatanodeInfoWithStorage[127.0.0.1:43593,DS-21c4b7b1-1127-402f-b86f-1701bea8bc66,DISK]] 2023-07-12 00:32:18,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34407] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 00:32:18,939 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(866): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1689121938488.b4ca85da7c922b089aeee8eea982fedf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:18,939 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1604): Closing b4ca85da7c922b089aeee8eea982fedf, disabling compactions & flushes 2023-07-12 00:32:18,939 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1626): Closing region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1689121938488.b4ca85da7c922b089aeee8eea982fedf. 2023-07-12 00:32:18,939 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1689121938488.b4ca85da7c922b089aeee8eea982fedf. 2023-07-12 00:32:18,939 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1714): Acquired close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1689121938488.b4ca85da7c922b089aeee8eea982fedf. after waiting 0 ms 2023-07-12 00:32:18,939 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1724): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1689121938488.b4ca85da7c922b089aeee8eea982fedf. 2023-07-12 00:32:18,939 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1838): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1689121938488.b4ca85da7c922b089aeee8eea982fedf. 2023-07-12 00:32:18,939 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1558): Region close journal for b4ca85da7c922b089aeee8eea982fedf: 2023-07-12 00:32:18,940 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(7675): creating {ENCODED => 591a2fafd71a4502f0c9d7c65c6de8cd, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1689121938488.591a2fafd71a4502f0c9d7c65c6de8cd.', STARTKEY => 'f', ENDKEY => 'g'}, tableDescriptor='testRegionAssignmentAfterMasterRecoveryDueToZKExpiry', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp 2023-07-12 00:32:18,941 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(866): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1689121938488.b36a989038618469dbf3c92ec0b243a1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:18,941 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(866): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1689121938488.700bb8bb9688a27326bcb98b7132a8d7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:18,941 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1604): Closing b36a989038618469dbf3c92ec0b243a1, disabling compactions & flushes 2023-07-12 00:32:18,941 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1604): Closing 700bb8bb9688a27326bcb98b7132a8d7, disabling compactions & flushes 2023-07-12 00:32:18,941 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1626): Closing region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1689121938488.b36a989038618469dbf3c92ec0b243a1. 2023-07-12 00:32:18,941 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1626): Closing region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1689121938488.700bb8bb9688a27326bcb98b7132a8d7. 2023-07-12 00:32:18,941 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1689121938488.b36a989038618469dbf3c92ec0b243a1. 2023-07-12 00:32:18,941 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1689121938488.700bb8bb9688a27326bcb98b7132a8d7. 2023-07-12 00:32:18,941 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1714): Acquired close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1689121938488.b36a989038618469dbf3c92ec0b243a1. after waiting 0 ms 2023-07-12 00:32:18,941 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1714): Acquired close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1689121938488.700bb8bb9688a27326bcb98b7132a8d7. after waiting 0 ms 2023-07-12 00:32:18,941 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1724): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1689121938488.b36a989038618469dbf3c92ec0b243a1. 2023-07-12 00:32:18,941 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1724): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1689121938488.700bb8bb9688a27326bcb98b7132a8d7. 2023-07-12 00:32:18,941 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1838): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1689121938488.b36a989038618469dbf3c92ec0b243a1. 2023-07-12 00:32:18,941 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1558): Region close journal for b36a989038618469dbf3c92ec0b243a1: 2023-07-12 00:32:18,941 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1838): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1689121938488.700bb8bb9688a27326bcb98b7132a8d7. 2023-07-12 00:32:18,942 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1558): Region close journal for 700bb8bb9688a27326bcb98b7132a8d7: 2023-07-12 00:32:18,942 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(7675): creating {ENCODED => 4bf803cbe0da319f9582ad75165052ed, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1689121938488.4bf803cbe0da319f9582ad75165052ed.', STARTKEY => 'g', ENDKEY => 'h'}, tableDescriptor='testRegionAssignmentAfterMasterRecoveryDueToZKExpiry', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp 2023-07-12 00:32:18,942 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(7675): creating {ENCODED => 6b5039c25691704ddb23db278cb8e6c5, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1689121938488.6b5039c25691704ddb23db278cb8e6c5.', STARTKEY => 'h', ENDKEY => 'i'}, tableDescriptor='testRegionAssignmentAfterMasterRecoveryDueToZKExpiry', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp 2023-07-12 00:32:18,963 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(866): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1689121938488.6b5039c25691704ddb23db278cb8e6c5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:18,964 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1604): Closing 6b5039c25691704ddb23db278cb8e6c5, disabling compactions & flushes 2023-07-12 00:32:18,964 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1626): Closing region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1689121938488.6b5039c25691704ddb23db278cb8e6c5. 2023-07-12 00:32:18,964 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1689121938488.6b5039c25691704ddb23db278cb8e6c5. 2023-07-12 00:32:18,964 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1714): Acquired close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1689121938488.6b5039c25691704ddb23db278cb8e6c5. after waiting 0 ms 2023-07-12 00:32:18,964 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1724): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1689121938488.6b5039c25691704ddb23db278cb8e6c5. 2023-07-12 00:32:18,964 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1838): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1689121938488.6b5039c25691704ddb23db278cb8e6c5. 2023-07-12 00:32:18,964 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1558): Region close journal for 6b5039c25691704ddb23db278cb8e6c5: 2023-07-12 00:32:18,964 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(7675): creating {ENCODED => d91c406970e2b1461f63e6cb82f78d8d, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1689121938488.d91c406970e2b1461f63e6cb82f78d8d.', STARTKEY => 'i', ENDKEY => 'j'}, tableDescriptor='testRegionAssignmentAfterMasterRecoveryDueToZKExpiry', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp 2023-07-12 00:32:18,973 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(866): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1689121938488.591a2fafd71a4502f0c9d7c65c6de8cd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:18,973 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(866): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1689121938488.4bf803cbe0da319f9582ad75165052ed.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:18,974 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1604): Closing 591a2fafd71a4502f0c9d7c65c6de8cd, disabling compactions & flushes 2023-07-12 00:32:18,974 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1604): Closing 4bf803cbe0da319f9582ad75165052ed, disabling compactions & flushes 2023-07-12 00:32:18,974 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1626): Closing region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1689121938488.591a2fafd71a4502f0c9d7c65c6de8cd. 2023-07-12 00:32:18,974 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1626): Closing region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1689121938488.4bf803cbe0da319f9582ad75165052ed. 2023-07-12 00:32:18,974 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1689121938488.591a2fafd71a4502f0c9d7c65c6de8cd. 2023-07-12 00:32:18,974 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1689121938488.4bf803cbe0da319f9582ad75165052ed. 2023-07-12 00:32:18,974 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1714): Acquired close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1689121938488.591a2fafd71a4502f0c9d7c65c6de8cd. after waiting 0 ms 2023-07-12 00:32:18,974 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1714): Acquired close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1689121938488.4bf803cbe0da319f9582ad75165052ed. after waiting 0 ms 2023-07-12 00:32:18,974 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1724): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1689121938488.591a2fafd71a4502f0c9d7c65c6de8cd. 2023-07-12 00:32:18,974 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1724): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1689121938488.4bf803cbe0da319f9582ad75165052ed. 2023-07-12 00:32:18,974 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1838): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1689121938488.591a2fafd71a4502f0c9d7c65c6de8cd. 2023-07-12 00:32:18,974 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1838): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1689121938488.4bf803cbe0da319f9582ad75165052ed. 2023-07-12 00:32:18,974 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1558): Region close journal for 591a2fafd71a4502f0c9d7c65c6de8cd: 2023-07-12 00:32:18,974 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-2] regionserver.HRegion(1558): Region close journal for 4bf803cbe0da319f9582ad75165052ed: 2023-07-12 00:32:18,975 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(7675): creating {ENCODED => 335cd564cec52505cf0fa421b43d2650, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1689121938488.335cd564cec52505cf0fa421b43d2650.', STARTKEY => 'j', ENDKEY => ''}, tableDescriptor='testRegionAssignmentAfterMasterRecoveryDueToZKExpiry', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/.tmp 2023-07-12 00:32:18,984 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(866): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1689121938488.d91c406970e2b1461f63e6cb82f78d8d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:18,984 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1604): Closing d91c406970e2b1461f63e6cb82f78d8d, disabling compactions & flushes 2023-07-12 00:32:18,984 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1626): Closing region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1689121938488.d91c406970e2b1461f63e6cb82f78d8d. 2023-07-12 00:32:18,984 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1689121938488.d91c406970e2b1461f63e6cb82f78d8d. 2023-07-12 00:32:18,984 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1714): Acquired close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1689121938488.d91c406970e2b1461f63e6cb82f78d8d. after waiting 0 ms 2023-07-12 00:32:18,984 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1724): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1689121938488.d91c406970e2b1461f63e6cb82f78d8d. 2023-07-12 00:32:18,984 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1838): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1689121938488.d91c406970e2b1461f63e6cb82f78d8d. 2023-07-12 00:32:18,984 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-1] regionserver.HRegion(1558): Region close journal for d91c406970e2b1461f63e6cb82f78d8d: 2023-07-12 00:32:18,986 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(866): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1689121938488.335cd564cec52505cf0fa421b43d2650.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:18,986 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1604): Closing 335cd564cec52505cf0fa421b43d2650, disabling compactions & flushes 2023-07-12 00:32:18,986 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1626): Closing region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1689121938488.335cd564cec52505cf0fa421b43d2650. 2023-07-12 00:32:18,987 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1689121938488.335cd564cec52505cf0fa421b43d2650. 2023-07-12 00:32:18,987 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1714): Acquired close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1689121938488.335cd564cec52505cf0fa421b43d2650. after waiting 0 ms 2023-07-12 00:32:18,987 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1724): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1689121938488.335cd564cec52505cf0fa421b43d2650. 2023-07-12 00:32:18,987 INFO [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1838): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1689121938488.335cd564cec52505cf0fa421b43d2650. 2023-07-12 00:32:18,987 DEBUG [RegionOpenAndInit-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-pool-0] regionserver.HRegion(1558): Region close journal for 335cd564cec52505cf0fa421b43d2650: 2023-07-12 00:32:18,989 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 00:32:18,991 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1689121938488.7a3cc4296899653225169eafd8b2a036.","families":{"info":[{"qualifier":"regioninfo","vlen":87,"tag":[],"timestamp":"1689121938990"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689121938990"}]},"ts":"1689121938990"} 2023-07-12 00:32:18,991 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1689121938488.02bb0fac148ca555b93632ec43604318.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121938990"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689121938990"}]},"ts":"1689121938990"} 2023-07-12 00:32:18,991 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1689121938488.491b73821b1517947c4dd95c17aad62c.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121938990"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689121938990"}]},"ts":"1689121938990"} 2023-07-12 00:32:18,991 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1689121938488.b4ca85da7c922b089aeee8eea982fedf.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121938990"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689121938990"}]},"ts":"1689121938990"} 2023-07-12 00:32:18,991 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1689121938488.b36a989038618469dbf3c92ec0b243a1.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121938990"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689121938990"}]},"ts":"1689121938990"} 2023-07-12 00:32:18,991 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1689121938488.700bb8bb9688a27326bcb98b7132a8d7.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121938990"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689121938990"}]},"ts":"1689121938990"} 2023-07-12 00:32:18,991 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1689121938488.6b5039c25691704ddb23db278cb8e6c5.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121938990"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689121938990"}]},"ts":"1689121938990"} 2023-07-12 00:32:18,991 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1689121938488.591a2fafd71a4502f0c9d7c65c6de8cd.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121938990"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689121938990"}]},"ts":"1689121938990"} 2023-07-12 00:32:18,991 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1689121938488.4bf803cbe0da319f9582ad75165052ed.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121938990"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689121938990"}]},"ts":"1689121938990"} 2023-07-12 00:32:18,991 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1689121938488.d91c406970e2b1461f63e6cb82f78d8d.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121938990"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689121938990"}]},"ts":"1689121938990"} 2023-07-12 00:32:18,991 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1689121938488.335cd564cec52505cf0fa421b43d2650.","families":{"info":[{"qualifier":"regioninfo","vlen":87,"tag":[],"timestamp":"1689121938990"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689121938990"}]},"ts":"1689121938990"} 2023-07-12 00:32:18,994 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 11 regions to meta. 2023-07-12 00:32:18,995 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 00:32:18,995 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689121938995"}]},"ts":"1689121938995"} 2023-07-12 00:32:18,996 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, state=ENABLING in hbase:meta 2023-07-12 00:32:18,998 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-07-12 00:32:19,003 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 00:32:19,003 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 00:32:19,003 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-07-12 00:32:19,003 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 00:32:19,003 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=7a3cc4296899653225169eafd8b2a036, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=491b73821b1517947c4dd95c17aad62c, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=02bb0fac148ca555b93632ec43604318, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=b4ca85da7c922b089aeee8eea982fedf, ASSIGN}, {pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=b36a989038618469dbf3c92ec0b243a1, ASSIGN}, {pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=700bb8bb9688a27326bcb98b7132a8d7, ASSIGN}, {pid=16, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=591a2fafd71a4502f0c9d7c65c6de8cd, ASSIGN}, {pid=17, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=4bf803cbe0da319f9582ad75165052ed, ASSIGN}, {pid=18, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=6b5039c25691704ddb23db278cb8e6c5, ASSIGN}, {pid=19, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=d91c406970e2b1461f63e6cb82f78d8d, ASSIGN}, {pid=20, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=335cd564cec52505cf0fa421b43d2650, ASSIGN}] 2023-07-12 00:32:19,006 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=d91c406970e2b1461f63e6cb82f78d8d, ASSIGN 2023-07-12 00:32:19,006 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=16, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=591a2fafd71a4502f0c9d7c65c6de8cd, ASSIGN 2023-07-12 00:32:19,006 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=4bf803cbe0da319f9582ad75165052ed, ASSIGN 2023-07-12 00:32:19,006 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=18, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=6b5039c25691704ddb23db278cb8e6c5, ASSIGN 2023-07-12 00:32:19,007 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=19, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=d91c406970e2b1461f63e6cb82f78d8d, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,39949,1689121937155; forceNewPlan=false, retain=false 2023-07-12 00:32:19,007 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=4bf803cbe0da319f9582ad75165052ed, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,33511,1689121937117; forceNewPlan=false, retain=false 2023-07-12 00:32:19,007 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=335cd564cec52505cf0fa421b43d2650, ASSIGN 2023-07-12 00:32:19,007 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=16, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=591a2fafd71a4502f0c9d7c65c6de8cd, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,41331,1689121938396; forceNewPlan=false, retain=false 2023-07-12 00:32:19,007 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=18, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=6b5039c25691704ddb23db278cb8e6c5, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,39949,1689121937155; forceNewPlan=false, retain=false 2023-07-12 00:32:19,007 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=700bb8bb9688a27326bcb98b7132a8d7, ASSIGN 2023-07-12 00:32:19,008 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=b36a989038618469dbf3c92ec0b243a1, ASSIGN 2023-07-12 00:32:19,008 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=700bb8bb9688a27326bcb98b7132a8d7, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,33511,1689121937117; forceNewPlan=false, retain=false 2023-07-12 00:32:19,008 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=02bb0fac148ca555b93632ec43604318, ASSIGN 2023-07-12 00:32:19,008 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=20, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=335cd564cec52505cf0fa421b43d2650, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,41331,1689121938396; forceNewPlan=false, retain=false 2023-07-12 00:32:19,008 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=b4ca85da7c922b089aeee8eea982fedf, ASSIGN 2023-07-12 00:32:19,009 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=b36a989038618469dbf3c92ec0b243a1, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,41331,1689121938396; forceNewPlan=false, retain=false 2023-07-12 00:32:19,009 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=02bb0fac148ca555b93632ec43604318, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,41331,1689121938396; forceNewPlan=false, retain=false 2023-07-12 00:32:19,009 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=491b73821b1517947c4dd95c17aad62c, ASSIGN 2023-07-12 00:32:19,009 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=7a3cc4296899653225169eafd8b2a036, ASSIGN 2023-07-12 00:32:19,009 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=b4ca85da7c922b089aeee8eea982fedf, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,39949,1689121937155; forceNewPlan=false, retain=false 2023-07-12 00:32:19,010 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=491b73821b1517947c4dd95c17aad62c, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,33511,1689121937117; forceNewPlan=false, retain=false 2023-07-12 00:32:19,010 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=7a3cc4296899653225169eafd8b2a036, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,39949,1689121937155; forceNewPlan=false, retain=false 2023-07-12 00:32:19,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34407] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 00:32:19,157 INFO [jenkins-hbase20:34407] balancer.BaseLoadBalancer(1545): Reassigned 11 regions. 11 retained the pre-restart assignment. 2023-07-12 00:32:19,167 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=335cd564cec52505cf0fa421b43d2650, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41331,1689121938396 2023-07-12 00:32:19,167 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=591a2fafd71a4502f0c9d7c65c6de8cd, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41331,1689121938396 2023-07-12 00:32:19,167 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=b36a989038618469dbf3c92ec0b243a1, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41331,1689121938396 2023-07-12 00:32:19,167 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1689121938488.591a2fafd71a4502f0c9d7c65c6de8cd.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121939167"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121939167"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121939167"}]},"ts":"1689121939167"} 2023-07-12 00:32:19,167 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1689121938488.b36a989038618469dbf3c92ec0b243a1.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121939167"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121939167"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121939167"}]},"ts":"1689121939167"} 2023-07-12 00:32:19,167 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=b4ca85da7c922b089aeee8eea982fedf, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39949,1689121937155 2023-07-12 00:32:19,167 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=02bb0fac148ca555b93632ec43604318, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41331,1689121938396 2023-07-12 00:32:19,167 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1689121938488.b4ca85da7c922b089aeee8eea982fedf.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121939167"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121939167"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121939167"}]},"ts":"1689121939167"} 2023-07-12 00:32:19,167 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1689121938488.335cd564cec52505cf0fa421b43d2650.","families":{"info":[{"qualifier":"regioninfo","vlen":87,"tag":[],"timestamp":"1689121939167"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121939167"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121939167"}]},"ts":"1689121939167"} 2023-07-12 00:32:19,168 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1689121938488.02bb0fac148ca555b93632ec43604318.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121939167"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121939167"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121939167"}]},"ts":"1689121939167"} 2023-07-12 00:32:19,169 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=16, state=RUNNABLE; OpenRegionProcedure 591a2fafd71a4502f0c9d7c65c6de8cd, server=jenkins-hbase20.apache.org,41331,1689121938396}] 2023-07-12 00:32:19,170 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=14, state=RUNNABLE; OpenRegionProcedure b36a989038618469dbf3c92ec0b243a1, server=jenkins-hbase20.apache.org,41331,1689121938396}] 2023-07-12 00:32:19,170 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=13, state=RUNNABLE; OpenRegionProcedure b4ca85da7c922b089aeee8eea982fedf, server=jenkins-hbase20.apache.org,39949,1689121937155}] 2023-07-12 00:32:19,171 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=20, state=RUNNABLE; OpenRegionProcedure 335cd564cec52505cf0fa421b43d2650, server=jenkins-hbase20.apache.org,41331,1689121938396}] 2023-07-12 00:32:19,171 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=7a3cc4296899653225169eafd8b2a036, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39949,1689121937155 2023-07-12 00:32:19,172 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1689121938488.7a3cc4296899653225169eafd8b2a036.","families":{"info":[{"qualifier":"regioninfo","vlen":87,"tag":[],"timestamp":"1689121939171"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121939171"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121939171"}]},"ts":"1689121939171"} 2023-07-12 00:32:19,173 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=12, state=RUNNABLE; OpenRegionProcedure 02bb0fac148ca555b93632ec43604318, server=jenkins-hbase20.apache.org,41331,1689121938396}] 2023-07-12 00:32:19,174 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=d91c406970e2b1461f63e6cb82f78d8d, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39949,1689121937155 2023-07-12 00:32:19,174 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1689121938488.d91c406970e2b1461f63e6cb82f78d8d.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121939174"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121939174"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121939174"}]},"ts":"1689121939174"} 2023-07-12 00:32:19,174 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=10, state=RUNNABLE; OpenRegionProcedure 7a3cc4296899653225169eafd8b2a036, server=jenkins-hbase20.apache.org,39949,1689121937155}] 2023-07-12 00:32:19,174 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=18 updating hbase:meta row=6b5039c25691704ddb23db278cb8e6c5, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39949,1689121937155 2023-07-12 00:32:19,174 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1689121938488.6b5039c25691704ddb23db278cb8e6c5.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121939174"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121939174"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121939174"}]},"ts":"1689121939174"} 2023-07-12 00:32:19,175 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=700bb8bb9688a27326bcb98b7132a8d7, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,33511,1689121937117 2023-07-12 00:32:19,175 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1689121938488.700bb8bb9688a27326bcb98b7132a8d7.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121939175"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121939175"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121939175"}]},"ts":"1689121939175"} 2023-07-12 00:32:19,175 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=19, state=RUNNABLE; OpenRegionProcedure d91c406970e2b1461f63e6cb82f78d8d, server=jenkins-hbase20.apache.org,39949,1689121937155}] 2023-07-12 00:32:19,177 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=4bf803cbe0da319f9582ad75165052ed, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,33511,1689121937117 2023-07-12 00:32:19,177 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1689121938488.4bf803cbe0da319f9582ad75165052ed.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121939177"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121939177"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121939177"}]},"ts":"1689121939177"} 2023-07-12 00:32:19,179 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=491b73821b1517947c4dd95c17aad62c, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,33511,1689121937117 2023-07-12 00:32:19,179 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1689121938488.491b73821b1517947c4dd95c17aad62c.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121939179"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689121939179"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689121939179"}]},"ts":"1689121939179"} 2023-07-12 00:32:19,179 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=18, state=RUNNABLE; OpenRegionProcedure 6b5039c25691704ddb23db278cb8e6c5, server=jenkins-hbase20.apache.org,39949,1689121937155}] 2023-07-12 00:32:19,180 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=15, state=RUNNABLE; OpenRegionProcedure 700bb8bb9688a27326bcb98b7132a8d7, server=jenkins-hbase20.apache.org,33511,1689121937117}] 2023-07-12 00:32:19,180 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=17, state=RUNNABLE; OpenRegionProcedure 4bf803cbe0da319f9582ad75165052ed, server=jenkins-hbase20.apache.org,33511,1689121937117}] 2023-07-12 00:32:19,182 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=11, state=RUNNABLE; OpenRegionProcedure 491b73821b1517947c4dd95c17aad62c, server=jenkins-hbase20.apache.org,33511,1689121937117}] 2023-07-12 00:32:19,321 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,41331,1689121938396 2023-07-12 00:32:19,321 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 00:32:19,324 INFO [RS-EventLoopGroup-24-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44860, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 00:32:19,329 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1689121938488.b4ca85da7c922b089aeee8eea982fedf. 2023-07-12 00:32:19,329 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b4ca85da7c922b089aeee8eea982fedf, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1689121938488.b4ca85da7c922b089aeee8eea982fedf.', STARTKEY => 'c', ENDKEY => 'd'} 2023-07-12 00:32:19,329 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1689121938488.591a2fafd71a4502f0c9d7c65c6de8cd. 2023-07-12 00:32:19,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 591a2fafd71a4502f0c9d7c65c6de8cd, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1689121938488.591a2fafd71a4502f0c9d7c65c6de8cd.', STARTKEY => 'f', ENDKEY => 'g'} 2023-07-12 00:32:19,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry b4ca85da7c922b089aeee8eea982fedf 2023-07-12 00:32:19,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1689121938488.b4ca85da7c922b089aeee8eea982fedf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:19,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 591a2fafd71a4502f0c9d7c65c6de8cd 2023-07-12 00:32:19,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for b4ca85da7c922b089aeee8eea982fedf 2023-07-12 00:32:19,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for b4ca85da7c922b089aeee8eea982fedf 2023-07-12 00:32:19,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1689121938488.591a2fafd71a4502f0c9d7c65c6de8cd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:19,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 591a2fafd71a4502f0c9d7c65c6de8cd 2023-07-12 00:32:19,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 591a2fafd71a4502f0c9d7c65c6de8cd 2023-07-12 00:32:19,331 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,33511,1689121937117 2023-07-12 00:32:19,331 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 00:32:19,333 INFO [StoreOpener-b4ca85da7c922b089aeee8eea982fedf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region b4ca85da7c922b089aeee8eea982fedf 2023-07-12 00:32:19,333 INFO [StoreOpener-591a2fafd71a4502f0c9d7c65c6de8cd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 591a2fafd71a4502f0c9d7c65c6de8cd 2023-07-12 00:32:19,334 INFO [RS-EventLoopGroup-22-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48620, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 00:32:19,335 DEBUG [StoreOpener-b4ca85da7c922b089aeee8eea982fedf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/b4ca85da7c922b089aeee8eea982fedf/info 2023-07-12 00:32:19,335 DEBUG [StoreOpener-b4ca85da7c922b089aeee8eea982fedf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/b4ca85da7c922b089aeee8eea982fedf/info 2023-07-12 00:32:19,335 DEBUG [StoreOpener-591a2fafd71a4502f0c9d7c65c6de8cd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/591a2fafd71a4502f0c9d7c65c6de8cd/info 2023-07-12 00:32:19,335 DEBUG [StoreOpener-591a2fafd71a4502f0c9d7c65c6de8cd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/591a2fafd71a4502f0c9d7c65c6de8cd/info 2023-07-12 00:32:19,335 INFO [StoreOpener-b4ca85da7c922b089aeee8eea982fedf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b4ca85da7c922b089aeee8eea982fedf columnFamilyName info 2023-07-12 00:32:19,336 INFO [StoreOpener-591a2fafd71a4502f0c9d7c65c6de8cd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 591a2fafd71a4502f0c9d7c65c6de8cd columnFamilyName info 2023-07-12 00:32:19,336 INFO [StoreOpener-b4ca85da7c922b089aeee8eea982fedf-1] regionserver.HStore(310): Store=b4ca85da7c922b089aeee8eea982fedf/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:19,336 INFO [StoreOpener-591a2fafd71a4502f0c9d7c65c6de8cd-1] regionserver.HStore(310): Store=591a2fafd71a4502f0c9d7c65c6de8cd/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:19,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/b4ca85da7c922b089aeee8eea982fedf 2023-07-12 00:32:19,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/591a2fafd71a4502f0c9d7c65c6de8cd 2023-07-12 00:32:19,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/b4ca85da7c922b089aeee8eea982fedf 2023-07-12 00:32:19,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/591a2fafd71a4502f0c9d7c65c6de8cd 2023-07-12 00:32:19,338 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1689121938488.491b73821b1517947c4dd95c17aad62c. 2023-07-12 00:32:19,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 491b73821b1517947c4dd95c17aad62c, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1689121938488.491b73821b1517947c4dd95c17aad62c.', STARTKEY => 'a', ENDKEY => 'b'} 2023-07-12 00:32:19,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 491b73821b1517947c4dd95c17aad62c 2023-07-12 00:32:19,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1689121938488.491b73821b1517947c4dd95c17aad62c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:19,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 491b73821b1517947c4dd95c17aad62c 2023-07-12 00:32:19,339 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 491b73821b1517947c4dd95c17aad62c 2023-07-12 00:32:19,340 INFO [StoreOpener-491b73821b1517947c4dd95c17aad62c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 491b73821b1517947c4dd95c17aad62c 2023-07-12 00:32:19,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for b4ca85da7c922b089aeee8eea982fedf 2023-07-12 00:32:19,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 591a2fafd71a4502f0c9d7c65c6de8cd 2023-07-12 00:32:19,341 DEBUG [StoreOpener-491b73821b1517947c4dd95c17aad62c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/491b73821b1517947c4dd95c17aad62c/info 2023-07-12 00:32:19,341 DEBUG [StoreOpener-491b73821b1517947c4dd95c17aad62c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/491b73821b1517947c4dd95c17aad62c/info 2023-07-12 00:32:19,342 INFO [StoreOpener-491b73821b1517947c4dd95c17aad62c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 491b73821b1517947c4dd95c17aad62c columnFamilyName info 2023-07-12 00:32:19,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/b4ca85da7c922b089aeee8eea982fedf/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:19,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/591a2fafd71a4502f0c9d7c65c6de8cd/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:19,342 INFO [StoreOpener-491b73821b1517947c4dd95c17aad62c-1] regionserver.HStore(310): Store=491b73821b1517947c4dd95c17aad62c/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:19,342 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened b4ca85da7c922b089aeee8eea982fedf; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60871517, jitterRate=-0.0929437130689621}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:19,342 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 591a2fafd71a4502f0c9d7c65c6de8cd; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69178650, jitterRate=0.030842214822769165}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:19,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for b4ca85da7c922b089aeee8eea982fedf: 2023-07-12 00:32:19,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 591a2fafd71a4502f0c9d7c65c6de8cd: 2023-07-12 00:32:19,343 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/491b73821b1517947c4dd95c17aad62c 2023-07-12 00:32:19,343 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1689121938488.591a2fafd71a4502f0c9d7c65c6de8cd., pid=21, masterSystemTime=1689121939321 2023-07-12 00:32:19,343 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1689121938488.b4ca85da7c922b089aeee8eea982fedf., pid=23, masterSystemTime=1689121939323 2023-07-12 00:32:19,345 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/491b73821b1517947c4dd95c17aad62c 2023-07-12 00:32:19,346 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1689121938488.591a2fafd71a4502f0c9d7c65c6de8cd. 2023-07-12 00:32:19,347 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1689121938488.591a2fafd71a4502f0c9d7c65c6de8cd. 2023-07-12 00:32:19,347 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1689121938488.335cd564cec52505cf0fa421b43d2650. 2023-07-12 00:32:19,348 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 335cd564cec52505cf0fa421b43d2650, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1689121938488.335cd564cec52505cf0fa421b43d2650.', STARTKEY => 'j', ENDKEY => ''} 2023-07-12 00:32:19,348 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=591a2fafd71a4502f0c9d7c65c6de8cd, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,41331,1689121938396 2023-07-12 00:32:19,348 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 335cd564cec52505cf0fa421b43d2650 2023-07-12 00:32:19,348 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1689121938488.b4ca85da7c922b089aeee8eea982fedf. 2023-07-12 00:32:19,348 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1689121938488.335cd564cec52505cf0fa421b43d2650.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:19,348 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1689121938488.591a2fafd71a4502f0c9d7c65c6de8cd.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121939348"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121939348"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121939348"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121939348"}]},"ts":"1689121939348"} 2023-07-12 00:32:19,348 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 335cd564cec52505cf0fa421b43d2650 2023-07-12 00:32:19,348 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 335cd564cec52505cf0fa421b43d2650 2023-07-12 00:32:19,348 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=b4ca85da7c922b089aeee8eea982fedf, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,39949,1689121937155 2023-07-12 00:32:19,348 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1689121938488.b4ca85da7c922b089aeee8eea982fedf. 2023-07-12 00:32:19,349 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1689121938488.d91c406970e2b1461f63e6cb82f78d8d. 2023-07-12 00:32:19,349 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1689121938488.b4ca85da7c922b089aeee8eea982fedf.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121939348"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121939348"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121939348"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121939348"}]},"ts":"1689121939348"} 2023-07-12 00:32:19,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d91c406970e2b1461f63e6cb82f78d8d, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1689121938488.d91c406970e2b1461f63e6cb82f78d8d.', STARTKEY => 'i', ENDKEY => 'j'} 2023-07-12 00:32:19,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry d91c406970e2b1461f63e6cb82f78d8d 2023-07-12 00:32:19,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1689121938488.d91c406970e2b1461f63e6cb82f78d8d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:19,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for d91c406970e2b1461f63e6cb82f78d8d 2023-07-12 00:32:19,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for d91c406970e2b1461f63e6cb82f78d8d 2023-07-12 00:32:19,351 INFO [StoreOpener-335cd564cec52505cf0fa421b43d2650-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 335cd564cec52505cf0fa421b43d2650 2023-07-12 00:32:19,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 491b73821b1517947c4dd95c17aad62c 2023-07-12 00:32:19,353 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=16 2023-07-12 00:32:19,353 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=16, state=SUCCESS; OpenRegionProcedure 591a2fafd71a4502f0c9d7c65c6de8cd, server=jenkins-hbase20.apache.org,41331,1689121938396 in 181 msec 2023-07-12 00:32:19,354 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=591a2fafd71a4502f0c9d7c65c6de8cd, ASSIGN in 350 msec 2023-07-12 00:32:19,355 INFO [StoreOpener-d91c406970e2b1461f63e6cb82f78d8d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region d91c406970e2b1461f63e6cb82f78d8d 2023-07-12 00:32:19,356 DEBUG [StoreOpener-335cd564cec52505cf0fa421b43d2650-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/335cd564cec52505cf0fa421b43d2650/info 2023-07-12 00:32:19,356 DEBUG [StoreOpener-335cd564cec52505cf0fa421b43d2650-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/335cd564cec52505cf0fa421b43d2650/info 2023-07-12 00:32:19,356 INFO [StoreOpener-335cd564cec52505cf0fa421b43d2650-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 335cd564cec52505cf0fa421b43d2650 columnFamilyName info 2023-07-12 00:32:19,357 INFO [StoreOpener-335cd564cec52505cf0fa421b43d2650-1] regionserver.HStore(310): Store=335cd564cec52505cf0fa421b43d2650/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:19,361 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=13 2023-07-12 00:32:19,361 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=13, state=SUCCESS; OpenRegionProcedure b4ca85da7c922b089aeee8eea982fedf, server=jenkins-hbase20.apache.org,39949,1689121937155 in 189 msec 2023-07-12 00:32:19,362 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=b4ca85da7c922b089aeee8eea982fedf, ASSIGN in 358 msec 2023-07-12 00:32:19,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/335cd564cec52505cf0fa421b43d2650 2023-07-12 00:32:19,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/491b73821b1517947c4dd95c17aad62c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:19,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/335cd564cec52505cf0fa421b43d2650 2023-07-12 00:32:19,369 DEBUG [StoreOpener-d91c406970e2b1461f63e6cb82f78d8d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/d91c406970e2b1461f63e6cb82f78d8d/info 2023-07-12 00:32:19,369 DEBUG [StoreOpener-d91c406970e2b1461f63e6cb82f78d8d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/d91c406970e2b1461f63e6cb82f78d8d/info 2023-07-12 00:32:19,369 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 491b73821b1517947c4dd95c17aad62c; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70541481, jitterRate=0.05114997923374176}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:19,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 491b73821b1517947c4dd95c17aad62c: 2023-07-12 00:32:19,370 INFO [StoreOpener-d91c406970e2b1461f63e6cb82f78d8d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d91c406970e2b1461f63e6cb82f78d8d columnFamilyName info 2023-07-12 00:32:19,370 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1689121938488.491b73821b1517947c4dd95c17aad62c., pid=31, masterSystemTime=1689121939331 2023-07-12 00:32:19,370 INFO [StoreOpener-d91c406970e2b1461f63e6cb82f78d8d-1] regionserver.HStore(310): Store=d91c406970e2b1461f63e6cb82f78d8d/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:19,373 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/d91c406970e2b1461f63e6cb82f78d8d 2023-07-12 00:32:19,373 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/d91c406970e2b1461f63e6cb82f78d8d 2023-07-12 00:32:19,374 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1689121938488.491b73821b1517947c4dd95c17aad62c. 2023-07-12 00:32:19,376 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1689121938488.491b73821b1517947c4dd95c17aad62c. 2023-07-12 00:32:19,376 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1689121938488.700bb8bb9688a27326bcb98b7132a8d7. 2023-07-12 00:32:19,376 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 700bb8bb9688a27326bcb98b7132a8d7, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1689121938488.700bb8bb9688a27326bcb98b7132a8d7.', STARTKEY => 'e', ENDKEY => 'f'} 2023-07-12 00:32:19,376 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 335cd564cec52505cf0fa421b43d2650 2023-07-12 00:32:19,377 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=491b73821b1517947c4dd95c17aad62c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,33511,1689121937117 2023-07-12 00:32:19,377 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 700bb8bb9688a27326bcb98b7132a8d7 2023-07-12 00:32:19,377 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1689121938488.700bb8bb9688a27326bcb98b7132a8d7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:19,377 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1689121938488.491b73821b1517947c4dd95c17aad62c.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121939377"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121939377"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121939377"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121939377"}]},"ts":"1689121939377"} 2023-07-12 00:32:19,377 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 700bb8bb9688a27326bcb98b7132a8d7 2023-07-12 00:32:19,377 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 700bb8bb9688a27326bcb98b7132a8d7 2023-07-12 00:32:19,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for d91c406970e2b1461f63e6cb82f78d8d 2023-07-12 00:32:19,380 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/335cd564cec52505cf0fa421b43d2650/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:19,380 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=11 2023-07-12 00:32:19,380 INFO [StoreOpener-700bb8bb9688a27326bcb98b7132a8d7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 700bb8bb9688a27326bcb98b7132a8d7 2023-07-12 00:32:19,380 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=11, state=SUCCESS; OpenRegionProcedure 491b73821b1517947c4dd95c17aad62c, server=jenkins-hbase20.apache.org,33511,1689121937117 in 197 msec 2023-07-12 00:32:19,380 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 335cd564cec52505cf0fa421b43d2650; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58922825, jitterRate=-0.12198148667812347}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:19,380 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 335cd564cec52505cf0fa421b43d2650: 2023-07-12 00:32:19,381 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/d91c406970e2b1461f63e6cb82f78d8d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:19,381 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened d91c406970e2b1461f63e6cb82f78d8d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71689540, jitterRate=0.0682573914527893}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:19,381 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1689121938488.335cd564cec52505cf0fa421b43d2650., pid=24, masterSystemTime=1689121939321 2023-07-12 00:32:19,381 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=491b73821b1517947c4dd95c17aad62c, ASSIGN in 377 msec 2023-07-12 00:32:19,381 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for d91c406970e2b1461f63e6cb82f78d8d: 2023-07-12 00:32:19,381 DEBUG [StoreOpener-700bb8bb9688a27326bcb98b7132a8d7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/700bb8bb9688a27326bcb98b7132a8d7/info 2023-07-12 00:32:19,381 DEBUG [StoreOpener-700bb8bb9688a27326bcb98b7132a8d7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/700bb8bb9688a27326bcb98b7132a8d7/info 2023-07-12 00:32:19,382 INFO [StoreOpener-700bb8bb9688a27326bcb98b7132a8d7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 700bb8bb9688a27326bcb98b7132a8d7 columnFamilyName info 2023-07-12 00:32:19,382 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1689121938488.d91c406970e2b1461f63e6cb82f78d8d., pid=27, masterSystemTime=1689121939323 2023-07-12 00:32:19,383 INFO [StoreOpener-700bb8bb9688a27326bcb98b7132a8d7-1] regionserver.HStore(310): Store=700bb8bb9688a27326bcb98b7132a8d7/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:19,383 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1689121938488.335cd564cec52505cf0fa421b43d2650. 2023-07-12 00:32:19,383 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1689121938488.335cd564cec52505cf0fa421b43d2650. 2023-07-12 00:32:19,383 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1689121938488.02bb0fac148ca555b93632ec43604318. 2023-07-12 00:32:19,383 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=335cd564cec52505cf0fa421b43d2650, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,41331,1689121938396 2023-07-12 00:32:19,383 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 02bb0fac148ca555b93632ec43604318, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1689121938488.02bb0fac148ca555b93632ec43604318.', STARTKEY => 'b', ENDKEY => 'c'} 2023-07-12 00:32:19,383 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1689121938488.335cd564cec52505cf0fa421b43d2650.","families":{"info":[{"qualifier":"regioninfo","vlen":87,"tag":[],"timestamp":"1689121939383"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121939383"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121939383"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121939383"}]},"ts":"1689121939383"} 2023-07-12 00:32:19,383 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1689121938488.d91c406970e2b1461f63e6cb82f78d8d. 2023-07-12 00:32:19,383 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 02bb0fac148ca555b93632ec43604318 2023-07-12 00:32:19,383 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1689121938488.d91c406970e2b1461f63e6cb82f78d8d. 2023-07-12 00:32:19,383 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1689121938488.02bb0fac148ca555b93632ec43604318.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:19,384 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1689121938488.6b5039c25691704ddb23db278cb8e6c5. 2023-07-12 00:32:19,384 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 02bb0fac148ca555b93632ec43604318 2023-07-12 00:32:19,383 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/700bb8bb9688a27326bcb98b7132a8d7 2023-07-12 00:32:19,384 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6b5039c25691704ddb23db278cb8e6c5, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1689121938488.6b5039c25691704ddb23db278cb8e6c5.', STARTKEY => 'h', ENDKEY => 'i'} 2023-07-12 00:32:19,384 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 02bb0fac148ca555b93632ec43604318 2023-07-12 00:32:19,384 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 6b5039c25691704ddb23db278cb8e6c5 2023-07-12 00:32:19,384 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=d91c406970e2b1461f63e6cb82f78d8d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,39949,1689121937155 2023-07-12 00:32:19,384 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1689121938488.6b5039c25691704ddb23db278cb8e6c5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:19,384 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/700bb8bb9688a27326bcb98b7132a8d7 2023-07-12 00:32:19,384 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1689121938488.d91c406970e2b1461f63e6cb82f78d8d.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121939384"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121939384"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121939384"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121939384"}]},"ts":"1689121939384"} 2023-07-12 00:32:19,384 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 6b5039c25691704ddb23db278cb8e6c5 2023-07-12 00:32:19,384 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 6b5039c25691704ddb23db278cb8e6c5 2023-07-12 00:32:19,385 INFO [StoreOpener-02bb0fac148ca555b93632ec43604318-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 02bb0fac148ca555b93632ec43604318 2023-07-12 00:32:19,386 INFO [StoreOpener-6b5039c25691704ddb23db278cb8e6c5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 6b5039c25691704ddb23db278cb8e6c5 2023-07-12 00:32:19,386 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=20 2023-07-12 00:32:19,387 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=20, state=SUCCESS; OpenRegionProcedure 335cd564cec52505cf0fa421b43d2650, server=jenkins-hbase20.apache.org,41331,1689121938396 in 214 msec 2023-07-12 00:32:19,387 DEBUG [StoreOpener-02bb0fac148ca555b93632ec43604318-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/02bb0fac148ca555b93632ec43604318/info 2023-07-12 00:32:19,387 DEBUG [StoreOpener-02bb0fac148ca555b93632ec43604318-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/02bb0fac148ca555b93632ec43604318/info 2023-07-12 00:32:19,387 DEBUG [StoreOpener-6b5039c25691704ddb23db278cb8e6c5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/6b5039c25691704ddb23db278cb8e6c5/info 2023-07-12 00:32:19,387 DEBUG [StoreOpener-6b5039c25691704ddb23db278cb8e6c5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/6b5039c25691704ddb23db278cb8e6c5/info 2023-07-12 00:32:19,387 INFO [StoreOpener-02bb0fac148ca555b93632ec43604318-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 02bb0fac148ca555b93632ec43604318 columnFamilyName info 2023-07-12 00:32:19,388 INFO [StoreOpener-6b5039c25691704ddb23db278cb8e6c5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6b5039c25691704ddb23db278cb8e6c5 columnFamilyName info 2023-07-12 00:32:19,388 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=19 2023-07-12 00:32:19,388 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 700bb8bb9688a27326bcb98b7132a8d7 2023-07-12 00:32:19,388 INFO [StoreOpener-02bb0fac148ca555b93632ec43604318-1] regionserver.HStore(310): Store=02bb0fac148ca555b93632ec43604318/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:19,388 INFO [StoreOpener-6b5039c25691704ddb23db278cb8e6c5-1] regionserver.HStore(310): Store=6b5039c25691704ddb23db278cb8e6c5/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:19,388 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=19, state=SUCCESS; OpenRegionProcedure d91c406970e2b1461f63e6cb82f78d8d, server=jenkins-hbase20.apache.org,39949,1689121937155 in 211 msec 2023-07-12 00:32:19,388 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=335cd564cec52505cf0fa421b43d2650, ASSIGN in 384 msec 2023-07-12 00:32:19,389 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/6b5039c25691704ddb23db278cb8e6c5 2023-07-12 00:32:19,389 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/02bb0fac148ca555b93632ec43604318 2023-07-12 00:32:19,389 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=d91c406970e2b1461f63e6cb82f78d8d, ASSIGN in 385 msec 2023-07-12 00:32:19,389 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/6b5039c25691704ddb23db278cb8e6c5 2023-07-12 00:32:19,389 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/02bb0fac148ca555b93632ec43604318 2023-07-12 00:32:19,390 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/700bb8bb9688a27326bcb98b7132a8d7/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:19,390 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 700bb8bb9688a27326bcb98b7132a8d7; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71430561, jitterRate=0.06439830362796783}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:19,390 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 700bb8bb9688a27326bcb98b7132a8d7: 2023-07-12 00:32:19,391 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1689121938488.700bb8bb9688a27326bcb98b7132a8d7., pid=29, masterSystemTime=1689121939331 2023-07-12 00:32:19,392 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 6b5039c25691704ddb23db278cb8e6c5 2023-07-12 00:32:19,392 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 02bb0fac148ca555b93632ec43604318 2023-07-12 00:32:19,392 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1689121938488.700bb8bb9688a27326bcb98b7132a8d7. 2023-07-12 00:32:19,393 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1689121938488.700bb8bb9688a27326bcb98b7132a8d7. 2023-07-12 00:32:19,393 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1689121938488.4bf803cbe0da319f9582ad75165052ed. 2023-07-12 00:32:19,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4bf803cbe0da319f9582ad75165052ed, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1689121938488.4bf803cbe0da319f9582ad75165052ed.', STARTKEY => 'g', ENDKEY => 'h'} 2023-07-12 00:32:19,393 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=700bb8bb9688a27326bcb98b7132a8d7, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,33511,1689121937117 2023-07-12 00:32:19,393 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1689121938488.700bb8bb9688a27326bcb98b7132a8d7.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121939393"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121939393"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121939393"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121939393"}]},"ts":"1689121939393"} 2023-07-12 00:32:19,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 4bf803cbe0da319f9582ad75165052ed 2023-07-12 00:32:19,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1689121938488.4bf803cbe0da319f9582ad75165052ed.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:19,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 4bf803cbe0da319f9582ad75165052ed 2023-07-12 00:32:19,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 4bf803cbe0da319f9582ad75165052ed 2023-07-12 00:32:19,395 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/6b5039c25691704ddb23db278cb8e6c5/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:19,395 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/02bb0fac148ca555b93632ec43604318/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:19,395 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 6b5039c25691704ddb23db278cb8e6c5; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74954115, jitterRate=0.11690334975719452}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:19,395 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 6b5039c25691704ddb23db278cb8e6c5: 2023-07-12 00:32:19,396 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 02bb0fac148ca555b93632ec43604318; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71972614, jitterRate=0.07247552275657654}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:19,396 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 02bb0fac148ca555b93632ec43604318: 2023-07-12 00:32:19,396 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1689121938488.6b5039c25691704ddb23db278cb8e6c5., pid=28, masterSystemTime=1689121939323 2023-07-12 00:32:19,396 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1689121938488.02bb0fac148ca555b93632ec43604318., pid=25, masterSystemTime=1689121939321 2023-07-12 00:32:19,397 INFO [StoreOpener-4bf803cbe0da319f9582ad75165052ed-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 4bf803cbe0da319f9582ad75165052ed 2023-07-12 00:32:19,399 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=18 updating hbase:meta row=6b5039c25691704ddb23db278cb8e6c5, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,39949,1689121937155 2023-07-12 00:32:19,399 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1689121938488.6b5039c25691704ddb23db278cb8e6c5.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121939399"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121939399"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121939399"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121939399"}]},"ts":"1689121939399"} 2023-07-12 00:32:19,399 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1689121938488.02bb0fac148ca555b93632ec43604318. 2023-07-12 00:32:19,399 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1689121938488.02bb0fac148ca555b93632ec43604318. 2023-07-12 00:32:19,399 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1689121938488.b36a989038618469dbf3c92ec0b243a1. 2023-07-12 00:32:19,400 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b36a989038618469dbf3c92ec0b243a1, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1689121938488.b36a989038618469dbf3c92ec0b243a1.', STARTKEY => 'd', ENDKEY => 'e'} 2023-07-12 00:32:19,400 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=15 2023-07-12 00:32:19,400 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry b36a989038618469dbf3c92ec0b243a1 2023-07-12 00:32:19,400 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1689121938488.b36a989038618469dbf3c92ec0b243a1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:19,400 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1689121938488.6b5039c25691704ddb23db278cb8e6c5. 2023-07-12 00:32:19,400 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for b36a989038618469dbf3c92ec0b243a1 2023-07-12 00:32:19,400 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for b36a989038618469dbf3c92ec0b243a1 2023-07-12 00:32:19,400 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=02bb0fac148ca555b93632ec43604318, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,41331,1689121938396 2023-07-12 00:32:19,400 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=15, state=SUCCESS; OpenRegionProcedure 700bb8bb9688a27326bcb98b7132a8d7, server=jenkins-hbase20.apache.org,33511,1689121937117 in 215 msec 2023-07-12 00:32:19,400 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1689121938488.02bb0fac148ca555b93632ec43604318.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121939400"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121939400"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121939400"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121939400"}]},"ts":"1689121939400"} 2023-07-12 00:32:19,400 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1689121938488.6b5039c25691704ddb23db278cb8e6c5. 2023-07-12 00:32:19,401 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1689121938488.7a3cc4296899653225169eafd8b2a036. 2023-07-12 00:32:19,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7a3cc4296899653225169eafd8b2a036, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1689121938488.7a3cc4296899653225169eafd8b2a036.', STARTKEY => '', ENDKEY => 'a'} 2023-07-12 00:32:19,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 7a3cc4296899653225169eafd8b2a036 2023-07-12 00:32:19,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1689121938488.7a3cc4296899653225169eafd8b2a036.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 00:32:19,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 7a3cc4296899653225169eafd8b2a036 2023-07-12 00:32:19,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 7a3cc4296899653225169eafd8b2a036 2023-07-12 00:32:19,402 DEBUG [StoreOpener-4bf803cbe0da319f9582ad75165052ed-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/4bf803cbe0da319f9582ad75165052ed/info 2023-07-12 00:32:19,402 DEBUG [StoreOpener-4bf803cbe0da319f9582ad75165052ed-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/4bf803cbe0da319f9582ad75165052ed/info 2023-07-12 00:32:19,402 INFO [StoreOpener-b36a989038618469dbf3c92ec0b243a1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region b36a989038618469dbf3c92ec0b243a1 2023-07-12 00:32:19,403 INFO [StoreOpener-4bf803cbe0da319f9582ad75165052ed-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4bf803cbe0da319f9582ad75165052ed columnFamilyName info 2023-07-12 00:32:19,403 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=700bb8bb9688a27326bcb98b7132a8d7, ASSIGN in 397 msec 2023-07-12 00:32:19,403 INFO [StoreOpener-4bf803cbe0da319f9582ad75165052ed-1] regionserver.HStore(310): Store=4bf803cbe0da319f9582ad75165052ed/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:19,404 INFO [StoreOpener-7a3cc4296899653225169eafd8b2a036-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 7a3cc4296899653225169eafd8b2a036 2023-07-12 00:32:19,404 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/4bf803cbe0da319f9582ad75165052ed 2023-07-12 00:32:19,405 DEBUG [StoreOpener-b36a989038618469dbf3c92ec0b243a1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/b36a989038618469dbf3c92ec0b243a1/info 2023-07-12 00:32:19,405 DEBUG [StoreOpener-b36a989038618469dbf3c92ec0b243a1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/b36a989038618469dbf3c92ec0b243a1/info 2023-07-12 00:32:19,405 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/4bf803cbe0da319f9582ad75165052ed 2023-07-12 00:32:19,405 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=28, resume processing ppid=18 2023-07-12 00:32:19,405 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=18, state=SUCCESS; OpenRegionProcedure 6b5039c25691704ddb23db278cb8e6c5, server=jenkins-hbase20.apache.org,39949,1689121937155 in 222 msec 2023-07-12 00:32:19,405 INFO [StoreOpener-b36a989038618469dbf3c92ec0b243a1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b36a989038618469dbf3c92ec0b243a1 columnFamilyName info 2023-07-12 00:32:19,405 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=12 2023-07-12 00:32:19,405 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=12, state=SUCCESS; OpenRegionProcedure 02bb0fac148ca555b93632ec43604318, server=jenkins-hbase20.apache.org,41331,1689121938396 in 229 msec 2023-07-12 00:32:19,405 DEBUG [StoreOpener-7a3cc4296899653225169eafd8b2a036-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/7a3cc4296899653225169eafd8b2a036/info 2023-07-12 00:32:19,405 DEBUG [StoreOpener-7a3cc4296899653225169eafd8b2a036-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/7a3cc4296899653225169eafd8b2a036/info 2023-07-12 00:32:19,406 INFO [StoreOpener-b36a989038618469dbf3c92ec0b243a1-1] regionserver.HStore(310): Store=b36a989038618469dbf3c92ec0b243a1/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:19,406 INFO [StoreOpener-7a3cc4296899653225169eafd8b2a036-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7a3cc4296899653225169eafd8b2a036 columnFamilyName info 2023-07-12 00:32:19,407 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=6b5039c25691704ddb23db278cb8e6c5, ASSIGN in 402 msec 2023-07-12 00:32:19,407 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=02bb0fac148ca555b93632ec43604318, ASSIGN in 402 msec 2023-07-12 00:32:19,407 INFO [StoreOpener-7a3cc4296899653225169eafd8b2a036-1] regionserver.HStore(310): Store=7a3cc4296899653225169eafd8b2a036/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 00:32:19,407 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/b36a989038618469dbf3c92ec0b243a1 2023-07-12 00:32:19,407 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/b36a989038618469dbf3c92ec0b243a1 2023-07-12 00:32:19,407 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/7a3cc4296899653225169eafd8b2a036 2023-07-12 00:32:19,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/7a3cc4296899653225169eafd8b2a036 2023-07-12 00:32:19,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 4bf803cbe0da319f9582ad75165052ed 2023-07-12 00:32:19,409 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for b36a989038618469dbf3c92ec0b243a1 2023-07-12 00:32:19,409 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/4bf803cbe0da319f9582ad75165052ed/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:19,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 7a3cc4296899653225169eafd8b2a036 2023-07-12 00:32:19,410 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 4bf803cbe0da319f9582ad75165052ed; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61147328, jitterRate=-0.08883380889892578}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:19,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 4bf803cbe0da319f9582ad75165052ed: 2023-07-12 00:32:19,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/b36a989038618469dbf3c92ec0b243a1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:19,411 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1689121938488.4bf803cbe0da319f9582ad75165052ed., pid=30, masterSystemTime=1689121939331 2023-07-12 00:32:19,411 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened b36a989038618469dbf3c92ec0b243a1; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72668653, jitterRate=0.08284731209278107}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:19,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/7a3cc4296899653225169eafd8b2a036/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 00:32:19,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for b36a989038618469dbf3c92ec0b243a1: 2023-07-12 00:32:19,411 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 7a3cc4296899653225169eafd8b2a036; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74282175, jitterRate=0.10689066350460052}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 00:32:19,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 7a3cc4296899653225169eafd8b2a036: 2023-07-12 00:32:19,412 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1689121938488.b36a989038618469dbf3c92ec0b243a1., pid=22, masterSystemTime=1689121939321 2023-07-12 00:32:19,412 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1689121938488.7a3cc4296899653225169eafd8b2a036., pid=26, masterSystemTime=1689121939323 2023-07-12 00:32:19,412 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1689121938488.4bf803cbe0da319f9582ad75165052ed. 2023-07-12 00:32:19,412 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1689121938488.4bf803cbe0da319f9582ad75165052ed. 2023-07-12 00:32:19,412 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=4bf803cbe0da319f9582ad75165052ed, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,33511,1689121937117 2023-07-12 00:32:19,413 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1689121938488.b36a989038618469dbf3c92ec0b243a1. 2023-07-12 00:32:19,413 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1689121938488.4bf803cbe0da319f9582ad75165052ed.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121939412"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121939412"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121939412"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121939412"}]},"ts":"1689121939412"} 2023-07-12 00:32:19,413 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1689121938488.b36a989038618469dbf3c92ec0b243a1. 2023-07-12 00:32:19,413 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=b36a989038618469dbf3c92ec0b243a1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,41331,1689121938396 2023-07-12 00:32:19,413 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1689121938488.b36a989038618469dbf3c92ec0b243a1.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":"1689121939413"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121939413"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121939413"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121939413"}]},"ts":"1689121939413"} 2023-07-12 00:32:19,413 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1689121938488.7a3cc4296899653225169eafd8b2a036. 2023-07-12 00:32:19,413 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1689121938488.7a3cc4296899653225169eafd8b2a036. 2023-07-12 00:32:19,413 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=7a3cc4296899653225169eafd8b2a036, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,39949,1689121937155 2023-07-12 00:32:19,414 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1689121938488.7a3cc4296899653225169eafd8b2a036.","families":{"info":[{"qualifier":"regioninfo","vlen":87,"tag":[],"timestamp":"1689121939413"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689121939413"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689121939413"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689121939413"}]},"ts":"1689121939413"} 2023-07-12 00:32:19,415 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=17 2023-07-12 00:32:19,415 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=17, state=SUCCESS; OpenRegionProcedure 4bf803cbe0da319f9582ad75165052ed, server=jenkins-hbase20.apache.org,33511,1689121937117 in 234 msec 2023-07-12 00:32:19,416 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=14 2023-07-12 00:32:19,416 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=14, state=SUCCESS; OpenRegionProcedure b36a989038618469dbf3c92ec0b243a1, server=jenkins-hbase20.apache.org,41331,1689121938396 in 245 msec 2023-07-12 00:32:19,416 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=4bf803cbe0da319f9582ad75165052ed, ASSIGN in 412 msec 2023-07-12 00:32:19,417 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=10 2023-07-12 00:32:19,417 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=10, state=SUCCESS; OpenRegionProcedure 7a3cc4296899653225169eafd8b2a036, server=jenkins-hbase20.apache.org,39949,1689121937155 in 241 msec 2023-07-12 00:32:19,417 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=b36a989038618469dbf3c92ec0b243a1, ASSIGN in 413 msec 2023-07-12 00:32:19,418 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-12 00:32:19,418 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=7a3cc4296899653225169eafd8b2a036, ASSIGN in 414 msec 2023-07-12 00:32:19,418 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 00:32:19,418 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689121939418"}]},"ts":"1689121939418"} 2023-07-12 00:32:19,420 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, state=ENABLED in hbase:meta 2023-07-12 00:32:19,421 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 00:32:19,422 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry in 932 msec 2023-07-12 00:32:19,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34407] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 00:32:19,600 INFO [Listener at localhost.localdomain/34481] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, procId: 9 completed 2023-07-12 00:32:19,600 INFO [Listener at localhost.localdomain/34481] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 00:32:19,604 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:39949-0x1007e974181002d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:19,604 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:33511-0x1007e974181002c, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:19,604 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:41331-0x1007e9741810032, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:19,604 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:19,605 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:19,604 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:19,605 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:19,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41331-0x1007e9741810032, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:19,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39949-0x1007e974181002d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:19,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33511-0x1007e974181002c, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:19,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:19,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:19,606 DEBUG [zk-event-processor-pool-0] master.ActiveMasterManager(186): No master available. Notifying waiting threads 2023-07-12 00:32:19,607 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:41331-0x1007e9741810032, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:19,607 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:39949-0x1007e974181002d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:19,607 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:19,607 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:33511-0x1007e974181002c, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:19,607 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:19,608 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:32:19,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 00:32:19,608 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,45037,1689121937044 from backup master directory 2023-07-12 00:32:19,609 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,45037,1689121937044 2023-07-12 00:32:19,609 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:32:19,609 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 00:32:19,609 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,45037,1689121937044 2023-07-12 00:32:19,627 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:19,650 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x2ebd8536 to 127.0.0.1:58965 with session timeout=1000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 00:32:19,655 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a07df22, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 00:32:19,656 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 00:32:19,656 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 00:32:19,656 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 00:32:19,663 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,34407,1689121937069 to hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,34407,1689121937069-dead as it is dead 2023-07-12 00:32:19,663 INFO [master/jenkins-hbase20:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,34407,1689121937069-dead/jenkins-hbase20.apache.org%2C34407%2C1689121937069.1689121937295 2023-07-12 00:32:19,664 WARN [IPC Server handler 2 on default port 44295] namenode.FSNamesystem(3291): DIR* NameSystem.internalReleaseLease: File /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,34407,1689121937069-dead/jenkins-hbase20.apache.org%2C34407%2C1689121937069.1689121937295 has not been closed. Lease recovery is in progress. RecoveryId = 1112 for block blk_1073741912_1090 2023-07-12 00:32:19,664 INFO [master/jenkins-hbase20:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Failed to recover lease, attempt=0 on file=hdfs://localhost.localdomain:44295/user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/MasterData/WALs/jenkins-hbase20.apache.org,34407,1689121937069-dead/jenkins-hbase20.apache.org%2C34407%2C1689121937069.1689121937295 after 1ms 2023-07-12 00:32:19,686 INFO [ReplicationExecutor-0] regionserver.ReplicationSourceManager$NodeFailoverWorker(712): Not transferring queue since we are shutting down 2023-07-12 00:32:19,705 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:19,705 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:19,706 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 00:32:19,706 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(159): ***** ABORTING master jenkins-hbase20.apache.org,34407,1689121937069: Test recovery from zk session expired ***** org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:197) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:32:19,706 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,34407,1689121937069' ***** 2023-07-12 00:32:19,706 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2311): STOPPED: Stopped by Listener at localhost.localdomain/34481 2023-07-12 00:32:19,707 WARN [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(402): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Unable to get data of znode /hbase/master org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:134) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:2131) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:330) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:394) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:145) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:324) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2824) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2816) at org.apache.hadoop.hbase.master.HMaster.abort(HMaster.java:2727) at org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:197) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:32:19,708 ERROR [Listener at localhost.localdomain/34481] zookeeper.ZKWatcher(701): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:134) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:2131) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:330) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:394) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:145) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:324) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2824) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2816) at org.apache.hadoop.hbase.master.HMaster.abort(HMaster.java:2727) at org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:197) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:32:19,708 DEBUG [Listener at localhost.localdomain/34481] master.ActiveMasterManager(335): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Failed delete of our master address node; KeeperErrorCode = Session expired for /hbase/master 2023-07-12 00:32:19,708 DEBUG [M:1;jenkins-hbase20:34407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@76f1951, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-12 00:32:19,708 INFO [M:1;jenkins-hbase20:34407] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,34407,1689121937069 2023-07-12 00:32:19,708 DEBUG [M:1;jenkins-hbase20:34407] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4163bcd9 to 127.0.0.1:58965 2023-07-12 00:32:19,708 DEBUG [M:1;jenkins-hbase20:34407] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:19,708 INFO [M:1;jenkins-hbase20:34407] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,34407,1689121937069; all regions closed. 2023-07-12 00:32:19,708 DEBUG [M:1;jenkins-hbase20:34407] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:19,708 INFO [Listener at localhost.localdomain/34481] hbase.Waiter(180): Waiting up to [5,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 00:32:19,709 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): ***** ABORTING master jenkins-hbase20.apache.org,45037,1689121937044: killAll ***** 2023-07-12 00:32:19,708 DEBUG [M:1;jenkins-hbase20:34407] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 00:32:19,709 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,45037,1689121937044' ***** 2023-07-12 00:32:19,709 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689121937406] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689121937406,5,FailOnTimeoutGroup] 2023-07-12 00:32:19,709 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 00:32:19,709 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2311): STOPPED: Stopped by Listener at localhost.localdomain/34481 2023-07-12 00:32:19,709 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689121937413] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689121937413,5,FailOnTimeoutGroup] 2023-07-12 00:32:19,709 DEBUG [M:1;jenkins-hbase20:34407] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 00:32:19,712 INFO [M:1;jenkins-hbase20:34407] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 00:32:19,712 DEBUG [M:0;jenkins-hbase20:45037] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7607b3e1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-12 00:32:19,712 INFO [M:1;jenkins-hbase20:34407] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 00:32:19,712 INFO [M:0;jenkins-hbase20:45037] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,45037,1689121937044 2023-07-12 00:32:19,712 DEBUG [M:0;jenkins-hbase20:45037] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ebd8536 to 127.0.0.1:58965 2023-07-12 00:32:19,712 INFO [M:1;jenkins-hbase20:34407] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-07-12 00:32:19,712 DEBUG [M:0;jenkins-hbase20:45037] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:19,712 DEBUG [M:1;jenkins-hbase20:34407] master.HMaster(1512): Stopping service threads 2023-07-12 00:32:19,712 INFO [M:0;jenkins-hbase20:45037] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,45037,1689121937044; all regions closed. 2023-07-12 00:32:19,712 DEBUG [M:0;jenkins-hbase20:45037] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:19,712 INFO [M:1;jenkins-hbase20:34407] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 00:32:19,712 INFO [M:0;jenkins-hbase20:45037] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-07-12 00:32:19,712 DEBUG [M:0;jenkins-hbase20:45037] master.HMaster(1512): Stopping service threads 2023-07-12 00:32:19,713 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:19,713 INFO [M:1;jenkins-hbase20:34407] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 00:32:19,713 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:19,713 DEBUG [Listener at localhost.localdomain/34481] master.HMaster(2711): Abort called but aborted=true, stopped=true 2023-07-12 00:32:19,713 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:39949-0x1007e974181002d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:19,713 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:33511-0x1007e974181002c, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:19,713 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:41331-0x1007e9741810032, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 00:32:19,714 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase20.apache.org,33511,1689121937117: killAll ***** 2023-07-12 00:32:19,714 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 00:32:19,714 ERROR [M:1;jenkins-hbase20:34407] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 00:32:19,714 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-12 00:32:19,714 INFO [M:1;jenkins-hbase20:34407] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=true 2023-07-12 00:32:19,714 DEBUG [M:0;jenkins-hbase20:45037] zookeeper.ZKUtil(398): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 00:32:19,714 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 00:32:19,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:19,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39949-0x1007e974181002d, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:19,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33511-0x1007e974181002c, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:19,715 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-12 00:32:19,714 WARN [M:0;jenkins-hbase20:45037] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 00:32:19,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41331-0x1007e9741810032, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 00:32:19,715 INFO [M:0;jenkins-hbase20:45037] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:45037 2023-07-12 00:32:19,716 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-12 00:32:19,716 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-12 00:32:19,717 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1192755200, "init": 524288000, "max": 2051014656, "used": 570692232 }, "NonHeapMemoryUsage": { "committed": 144072704, "init": 2555904, "max": -1, "used": 141460352 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 0, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 45102, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 1, "ProcessCallTime_num_ops": 167, "ProcessCallTime_min": 0, "ProcessCallTime_max": 23, "ProcessCallTime_mean": 2, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 1, "ProcessCallTime_75th_percentile": 3, "ProcessCallTime_90th_percentile": 4, "ProcessCallTime_95th_percentile": 9, "ProcessCallTime_98th_percentile": 17, "ProcessCallTime_99th_percentile": 23, "ProcessCallTime_99.9th_percentile": 23, "ProcessCallTime_TimeRangeCount_0-1": 30, "ProcessCallTime_TimeRangeCount_1-3": 12, "ProcessCallTime_TimeRangeCount_3-10": 9, "ProcessCallTime_TimeRangeCount_10-30": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 167, "QueueCallTime_min": 0, "QueueCallTime_max": 6, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 1, "QueueCallTime_95th_percentile": 2, "QueueCallTime_98th_percentile": 3, "QueueCallTime_99th_percentile": 6, "QueueCallTime_99.9th_percentile": 6, "QueueCallTime_TimeRangeCount_0-1": 49, "QueueCallTime_TimeRangeCount_1-3": 3, "QueueCallTime_TimeRangeCount_3-10": 1, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 167, "TotalCallTime_min": 0, "TotalCallTime_max": 23, "TotalCallTime_mean": 2, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 1, "TotalCallTime_75th_percentile": 4, "TotalCallTime_90th_percentile": 7, "TotalCallTime_95th_percentile": 9, "TotalCallTime_98th_percentile": 17, "TotalCallTime_99th_percentile": 23, "TotalCallTime_99.9th_percentile": 23, "TotalCallTime_TimeRangeCount_0-1": 27, "TotalCallTime_TimeRangeCount_1-3": 12, "TotalCallTime_TimeRangeCount_3-10": 12, "TotalCallTime_TimeRangeCount_10-30": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 167, "ResponseSize_min": 0, "ResponseSize_max": 5990, "ResponseSize_mean": 208, "ResponseSize_25th_percentile": 1, "ResponseSize_median": 2, "ResponseSize_75th_percentile": 5, "ResponseSize_90th_percentile": 769, "ResponseSize_95th_percentile": 2683, "ResponseSize_98th_percentile": 4667, "ResponseSize_99th_percentile": 5328, "ResponseSize_99.9th_percentile": 5923, "ResponseSize_SizeRangeCount_0-10": 43, "ResponseSize_SizeRangeCount_10-100": 3, "ResponseSize_SizeRangeCount_100-1000": 7, "exceptions.UnknownScannerException": 0, "exceptions": 2, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 1, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 167, "RequestSize_min": 80, "RequestSize_max": 2899, "RequestSize_mean": 269, "RequestSize_25th_percentile": 145, "RequestSize_median": 176, "RequestSize_75th_percentile": 339, "RequestSize_90th_percentile": 1009, "RequestSize_95th_percentile": 1954, "RequestSize_98th_percentile": 2521, "RequestSize_99th_percentile": 2710, "RequestSize_99.9th_percentile": 2880, "RequestSize_SizeRangeCount_10-100": 5, "RequestSize_SizeRangeCount_100-1000": 48, "sentBytes": 28192 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:58965", "tag.serverName": "jenkins-hbase20.apache.org,42069,1689121928319", "tag.clusterId": "b855a1de-f551-45f5-8b9b-f7b8ff57b75f", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "regionCount": 4, "storeCount": 6, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 2, "maxStoreFileCount": 1, "memStoreSize": 78, "storeFileSize": 21223, "maxStoreFileAge": 385, "minStoreFileAge": 333, "avgStoreFileAge": 359, "numReferenceFiles": 0, "regionServerStartTime": 1689121928319, "averageRegionSize": 5325, "storeFileIndexSize": 920, "staticIndexSize": 290, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 79133, "localBytesRead": 79133, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819794032, "blockCacheCount": 2, "blockCacheDataBlockCount": 2, "blockCacheSize": 611856, "blockCacheCountHitPercent": 50.0, "blockCacheExpressHitPercent": 50.0, "l1CacheSize": 611856, "l1CacheFreeSize": 819794032, "l1CacheCount": 2, "l1CacheEvictionCount": 0, "l1CacheHitCount": 2, "l1CacheMissCount": 2, "l1CacheHitRatio": 0.5, "l1CacheMissRatio": 0.5, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 1.2, "writeRequestRatePerSecond": 0.4, "ByteBuffAllocatorHeapAllocationBytes": 147754, "ByteBuffAllocatorPoolAllocationBytes": 599040, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 12, "totalRowActionRequestCount": 8, "readRequestCount": 6, "filteredReadRequestCount": 0, "writeRequestCount": 2, "rpcGetRequestCount": 7, "rpcFullScanRequestCount": 0, "rpcScanRequestCount": 3, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 2, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 2, "blockCacheHitCountPrimary": 2, "blockCacheHitCachingCount": 2, "blockCacheMissCount": 2, "blockCacheMissCountPrimary": 2, "blockCacheMissCachingCount": 2, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 2, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 2, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 74, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 11438, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 3, "ScanTime_min": 0, "ScanTime_max": 0, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 0, "ScanTime_75th_percentile": 0, "ScanTime_90th_percentile": 0, "ScanTime_95th_percentile": 0, "ScanTime_98th_percentile": 0, "ScanTime_99th_percentile": 0, "ScanTime_99.9th_percentile": 0, "ScanTime_TimeRangeCount_0-1": 3, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 2, "Put_min": 1, "Put_max": 3, "Put_mean": 2, "Put_25th_percentile": 1, "Put_median": 2, "Put_75th_percentile": 3, "Put_90th_percentile": 3, "Put_95th_percentile": 3, "Put_98th_percentile": 3, "Put_99th_percentile": 3, "Put_99.9th_percentile": 3, "Put_TimeRangeCount_0-1": 1, "Put_TimeRangeCount_1-3": 1, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 1, "FlushMemstoreSize_min": 11438, "FlushMemstoreSize_max": 11438, "FlushMemstoreSize_mean": 11438, "FlushMemstoreSize_25th_percentile": 11438, "FlushMemstoreSize_median": 11438, "FlushMemstoreSize_75th_percentile": 11438, "FlushMemstoreSize_90th_percentile": 11438, "FlushMemstoreSize_95th_percentile": 11438, "FlushMemstoreSize_98th_percentile": 11438, "FlushMemstoreSize_99th_percentile": 11438, "FlushMemstoreSize_99.9th_percentile": 11438, "FlushMemstoreSize_SizeRangeCount_100-1000": 1, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 6, "Get_min": 0, "Get_max": 23, "Get_mean": 4, "Get_25th_percentile": 0, "Get_median": 1, "Get_75th_percentile": 2, "Get_90th_percentile": 23, "Get_95th_percentile": 23, "Get_98th_percentile": 23, "Get_99th_percentile": 23, "Get_99.9th_percentile": 23, "Get_TimeRangeCount_0-1": 4, "Get_TimeRangeCount_1-3": 1, "Get_TimeRangeCount_10-30": 1, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 21223, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 1, "FlushTime_min": 154, "FlushTime_max": 154, "FlushTime_mean": 154, "FlushTime_25th_percentile": 154, "FlushTime_median": 154, "FlushTime_75th_percentile": 154, "FlushTime_90th_percentile": 154, "FlushTime_95th_percentile": 154, "FlushTime_98th_percentile": 154, "FlushTime_99th_percentile": 154, "FlushTime_99.9th_percentile": 154, "FlushTime_TimeRangeCount_100-300": 1, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 3, "ScanSize_min": 86, "ScanSize_max": 985, "ScanSize_mean": 685, "ScanSize_25th_percentile": 86, "ScanSize_median": 438, "ScanSize_75th_percentile": 711, "ScanSize_90th_percentile": 875, "ScanSize_95th_percentile": 930, "ScanSize_98th_percentile": 963, "ScanSize_99th_percentile": 974, "ScanSize_99.9th_percentile": 983, "ScanSize_SizeRangeCount_10-100": 1, "ScanSize_SizeRangeCount_100-1000": 2, "slowGetCount": 0, "flushedMemstoreBytes": 11438, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 1, "FlushOutputSize_min": 21223, "FlushOutputSize_max": 21223, "FlushOutputSize_mean": 21223, "FlushOutputSize_25th_percentile": 21223, "FlushOutputSize_median": 21223, "FlushOutputSize_75th_percentile": 21223, "FlushOutputSize_90th_percentile": 21223, "FlushOutputSize_95th_percentile": 21223, "FlushOutputSize_98th_percentile": 21223, "FlushOutputSize_99th_percentile": 21223, "FlushOutputSize_99.9th_percentile": 21223, "FlushOutputSize_SizeRangeCount_100-1000": 1, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 93, "ServerReadQueryPerSecond_mean_rate": 3.0241271332370667, "ServerReadQueryPerSecond_1min_rate": 2.81760632426473, "ServerReadQueryPerSecond_5min_rate": 2.6383756459467356, "ServerReadQueryPerSecond_15min_rate": 2.6124191975460884, "ServerWriteQueryPerSecond_count": 74, "ServerWriteQueryPerSecond_mean_rate": 2.406294348075967, "ServerWriteQueryPerSecond_1min_rate": 1.902961977749651, "ServerWriteQueryPerSecond_5min_rate": 1.5054020168326836, "ServerWriteQueryPerSecond_15min_rate": 1.4354140700006486 } ] } 2023-07-12 00:32:19,717 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34407] master.MasterRpcServices(612): jenkins-hbase20.apache.org,33511,1689121937117 reported a fatal error: ***** ABORTING region server jenkins-hbase20.apache.org,33511,1689121937117: killAll ***** 2023-07-12 00:32:19,718 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,33511,1689121937117' ***** 2023-07-12 00:32:19,718 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2311): STOPPED: killAll 2023-07-12 00:32:19,718 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase20.apache.org,39949,1689121937155: killAll ***** 2023-07-12 00:32:19,718 INFO [RS:0;jenkins-hbase20:33511] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 00:32:19,718 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 00:32:19,718 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 00:32:19,718 INFO [RS:0;jenkins-hbase20:33511] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-12 00:32:19,718 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-12 00:32:19,719 INFO [RS:0;jenkins-hbase20:33511] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-12 00:32:19,719 INFO [RS:0;jenkins-hbase20:33511] regionserver.HRegionServer(3305): Received CLOSE for 700bb8bb9688a27326bcb98b7132a8d7 2023-07-12 00:32:19,719 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-12 00:32:19,720 INFO [RS:0;jenkins-hbase20:33511] regionserver.HRegionServer(3305): Received CLOSE for 491b73821b1517947c4dd95c17aad62c 2023-07-12 00:32:19,720 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-12 00:32:19,721 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 700bb8bb9688a27326bcb98b7132a8d7, disabling compactions & flushes 2023-07-12 00:32:19,720 INFO [RS:0;jenkins-hbase20:33511] regionserver.HRegionServer(3305): Received CLOSE for 4bf803cbe0da319f9582ad75165052ed 2023-07-12 00:32:19,721 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1689121938488.700bb8bb9688a27326bcb98b7132a8d7. 2023-07-12 00:32:19,721 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-12 00:32:19,721 INFO [RS:0;jenkins-hbase20:33511] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,33511,1689121937117 2023-07-12 00:32:19,721 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1689121938488.700bb8bb9688a27326bcb98b7132a8d7. 2023-07-12 00:32:19,721 DEBUG [RS:0;jenkins-hbase20:33511] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x288369ca to 127.0.0.1:58965 2023-07-12 00:32:19,721 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1689121938488.700bb8bb9688a27326bcb98b7132a8d7. after waiting 0 ms 2023-07-12 00:32:19,721 DEBUG [RS:0;jenkins-hbase20:33511] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:19,721 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1689121938488.700bb8bb9688a27326bcb98b7132a8d7. 2023-07-12 00:32:19,721 INFO [RS:0;jenkins-hbase20:33511] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-12 00:32:19,721 DEBUG [RS:0;jenkins-hbase20:33511] regionserver.HRegionServer(1478): Online Regions={700bb8bb9688a27326bcb98b7132a8d7=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1689121938488.700bb8bb9688a27326bcb98b7132a8d7., 491b73821b1517947c4dd95c17aad62c=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1689121938488.491b73821b1517947c4dd95c17aad62c., 4bf803cbe0da319f9582ad75165052ed=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1689121938488.4bf803cbe0da319f9582ad75165052ed.} 2023-07-12 00:32:19,721 DEBUG [RS:0;jenkins-hbase20:33511] regionserver.HRegionServer(1504): Waiting on 491b73821b1517947c4dd95c17aad62c, 4bf803cbe0da319f9582ad75165052ed, 700bb8bb9688a27326bcb98b7132a8d7 2023-07-12 00:32:19,721 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1689121938488.700bb8bb9688a27326bcb98b7132a8d7. 2023-07-12 00:32:19,721 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 700bb8bb9688a27326bcb98b7132a8d7: 2023-07-12 00:32:19,722 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1689121938488.700bb8bb9688a27326bcb98b7132a8d7. 2023-07-12 00:32:19,722 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 491b73821b1517947c4dd95c17aad62c, disabling compactions & flushes 2023-07-12 00:32:19,722 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1689121938488.491b73821b1517947c4dd95c17aad62c. 2023-07-12 00:32:19,723 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1689121938488.491b73821b1517947c4dd95c17aad62c. 2023-07-12 00:32:19,723 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1689121938488.491b73821b1517947c4dd95c17aad62c. after waiting 0 ms 2023-07-12 00:32:19,723 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1192755200, "init": 524288000, "max": 2051014656, "used": 570819784 }, "NonHeapMemoryUsage": { "committed": 144072704, "init": 2555904, "max": -1, "used": 141461248 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 0, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 45102, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 1, "ProcessCallTime_num_ops": 167, "ProcessCallTime_min": 0, "ProcessCallTime_max": 23, "ProcessCallTime_mean": 2, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 1, "ProcessCallTime_75th_percentile": 3, "ProcessCallTime_90th_percentile": 4, "ProcessCallTime_95th_percentile": 9, "ProcessCallTime_98th_percentile": 17, "ProcessCallTime_99th_percentile": 23, "ProcessCallTime_99.9th_percentile": 23, "ProcessCallTime_TimeRangeCount_0-1": 30, "ProcessCallTime_TimeRangeCount_1-3": 12, "ProcessCallTime_TimeRangeCount_3-10": 9, "ProcessCallTime_TimeRangeCount_10-30": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 167, "QueueCallTime_min": 0, "QueueCallTime_max": 6, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 1, "QueueCallTime_95th_percentile": 2, "QueueCallTime_98th_percentile": 3, "QueueCallTime_99th_percentile": 6, "QueueCallTime_99.9th_percentile": 6, "QueueCallTime_TimeRangeCount_0-1": 49, "QueueCallTime_TimeRangeCount_1-3": 3, "QueueCallTime_TimeRangeCount_3-10": 1, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 167, "TotalCallTime_min": 0, "TotalCallTime_max": 23, "TotalCallTime_mean": 2, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 1, "TotalCallTime_75th_percentile": 4, "TotalCallTime_90th_percentile": 7, "TotalCallTime_95th_percentile": 9, "TotalCallTime_98th_percentile": 17, "TotalCallTime_99th_percentile": 23, "TotalCallTime_99.9th_percentile": 23, "TotalCallTime_TimeRangeCount_0-1": 27, "TotalCallTime_TimeRangeCount_1-3": 12, "TotalCallTime_TimeRangeCount_3-10": 12, "TotalCallTime_TimeRangeCount_10-30": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 167, "ResponseSize_min": 0, "ResponseSize_max": 5990, "ResponseSize_mean": 208, "ResponseSize_25th_percentile": 1, "ResponseSize_median": 2, "ResponseSize_75th_percentile": 5, "ResponseSize_90th_percentile": 769, "ResponseSize_95th_percentile": 2683, "ResponseSize_98th_percentile": 4667, "ResponseSize_99th_percentile": 5328, "ResponseSize_99.9th_percentile": 5923, "ResponseSize_SizeRangeCount_0-10": 43, "ResponseSize_SizeRangeCount_10-100": 3, "ResponseSize_SizeRangeCount_100-1000": 7, "exceptions.UnknownScannerException": 0, "exceptions": 2, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 1, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 167, "RequestSize_min": 80, "RequestSize_max": 2899, "RequestSize_mean": 269, "RequestSize_25th_percentile": 145, "RequestSize_median": 176, "RequestSize_75th_percentile": 339, "RequestSize_90th_percentile": 1009, "RequestSize_95th_percentile": 1954, "RequestSize_98th_percentile": 2521, "RequestSize_99th_percentile": 2710, "RequestSize_99.9th_percentile": 2880, "RequestSize_SizeRangeCount_10-100": 5, "RequestSize_SizeRangeCount_100-1000": 48, "sentBytes": 28192 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:58965", "tag.serverName": "jenkins-hbase20.apache.org,42069,1689121928319", "tag.clusterId": "b855a1de-f551-45f5-8b9b-f7b8ff57b75f", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "regionCount": 4, "storeCount": 6, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 2, "maxStoreFileCount": 1, "memStoreSize": 78, "storeFileSize": 21223, "maxStoreFileAge": 385, "minStoreFileAge": 333, "avgStoreFileAge": 359, "numReferenceFiles": 0, "regionServerStartTime": 1689121928319, "averageRegionSize": 5325, "storeFileIndexSize": 920, "staticIndexSize": 290, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 79133, "localBytesRead": 79133, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819794032, "blockCacheCount": 2, "blockCacheDataBlockCount": 2, "blockCacheSize": 611856, "blockCacheCountHitPercent": 50.0, "blockCacheExpressHitPercent": 50.0, "l1CacheSize": 611856, "l1CacheFreeSize": 819794032, "l1CacheCount": 2, "l1CacheEvictionCount": 0, "l1CacheHitCount": 2, "l1CacheMissCount": 2, "l1CacheHitRatio": 0.5, "l1CacheMissRatio": 0.5, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 1.2, "writeRequestRatePerSecond": 0.4, "ByteBuffAllocatorHeapAllocationBytes": 147754, "ByteBuffAllocatorPoolAllocationBytes": 599040, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 12, "totalRowActionRequestCount": 8, "readRequestCount": 6, "filteredReadRequestCount": 0, "writeRequestCount": 2, "rpcGetRequestCount": 7, "rpcFullScanRequestCount": 0, "rpcScanRequestCount": 3, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 2, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 2, "blockCacheHitCountPrimary": 2, "blockCacheHitCachingCount": 2, "blockCacheMissCount": 2, "blockCacheMissCountPrimary": 2, "blockCacheMissCachingCount": 2, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 2, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 2, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 74, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 11438, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 3, "ScanTime_min": 0, "ScanTime_max": 0, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 0, "ScanTime_75th_percentile": 0, "ScanTime_90th_percentile": 0, "ScanTime_95th_percentile": 0, "ScanTime_98th_percentile": 0, "ScanTime_99th_percentile": 0, "ScanTime_99.9th_percentile": 0, "ScanTime_TimeRangeCount_0-1": 3, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 2, "Put_min": 1, "Put_max": 3, "Put_mean": 2, "Put_25th_percentile": 1, "Put_median": 2, "Put_75th_percentile": 3, "Put_90th_percentile": 3, "Put_95th_percentile": 3, "Put_98th_percentile": 3, "Put_99th_percentile": 3, "Put_99.9th_percentile": 3, "Put_TimeRangeCount_0-1": 1, "Put_TimeRangeCount_1-3": 1, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 1, "FlushMemstoreSize_min": 11438, "FlushMemstoreSize_max": 11438, "FlushMemstoreSize_mean": 11438, "FlushMemstoreSize_25th_percentile": 11438, "FlushMemstoreSize_median": 11438, "FlushMemstoreSize_75th_percentile": 11438, "FlushMemstoreSize_90th_percentile": 11438, "FlushMemstoreSize_95th_percentile": 11438, "FlushMemstoreSize_98th_percentile": 11438, "FlushMemstoreSize_99th_percentile": 11438, "FlushMemstoreSize_99.9th_percentile": 11438, "FlushMemstoreSize_SizeRangeCount_100-1000": 1, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 6, "Get_min": 0, "Get_max": 23, "Get_mean": 4, "Get_25th_percentile": 0, "Get_median": 1, "Get_75th_percentile": 2, "Get_90th_percentile": 23, "Get_95th_percentile": 23, "Get_98th_percentile": 23, "Get_99th_percentile": 23, "Get_99.9th_percentile": 23, "Get_TimeRangeCount_0-1": 4, "Get_TimeRangeCount_1-3": 1, "Get_TimeRangeCount_10-30": 1, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 21223, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 1, "FlushTime_min": 154, "FlushTime_max": 154, "FlushTime_mean": 154, "FlushTime_25th_percentile": 154, "FlushTime_median": 154, "FlushTime_75th_percentile": 154, "FlushTime_90th_percentile": 154, "FlushTime_95th_percentile": 154, "FlushTime_98th_percentile": 154, "FlushTime_99th_percentile": 154, "FlushTime_99.9th_percentile": 154, "FlushTime_TimeRangeCount_100-300": 1, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 3, "ScanSize_min": 86, "ScanSize_max": 985, "ScanSize_mean": 685, "ScanSize_25th_percentile": 86, "ScanSize_median": 438, "ScanSize_75th_percentile": 711, "ScanSize_90th_percentile": 875, "ScanSize_95th_percentile": 930, "ScanSize_98th_percentile": 963, "ScanSize_99th_percentile": 974, "ScanSize_99.9th_percentile": 983, "ScanSize_SizeRangeCount_10-100": 1, "ScanSize_SizeRangeCount_100-1000": 2, "slowGetCount": 0, "flushedMemstoreBytes": 11438, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 1, "FlushOutputSize_min": 21223, "FlushOutputSize_max": 21223, "FlushOutputSize_mean": 21223, "FlushOutputSize_25th_percentile": 21223, "FlushOutputSize_median": 21223, "FlushOutputSize_75th_percentile": 21223, "FlushOutputSize_90th_percentile": 21223, "FlushOutputSize_95th_percentile": 21223, "FlushOutputSize_98th_percentile": 21223, "FlushOutputSize_99th_percentile": 21223, "FlushOutputSize_99.9th_percentile": 21223, "FlushOutputSize_SizeRangeCount_100-1000": 1, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 93, "ServerReadQueryPerSecond_mean_rate": 3.0241271332370667, "ServerReadQueryPerSecond_1min_rate": 2.81760632426473, "ServerReadQueryPerSecond_5min_rate": 2.6383756459467356, "ServerReadQueryPerSecond_15min_rate": 2.6124191975460884, "ServerWriteQueryPerSecond_count": 74, "ServerWriteQueryPerSecond_mean_rate": 2.406294348075967, "ServerWriteQueryPerSecond_1min_rate": 1.902961977749651, "ServerWriteQueryPerSecond_5min_rate": 1.5054020168326836, "ServerWriteQueryPerSecond_15min_rate": 1.4354140700006486 } ] } 2023-07-12 00:32:19,723 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1689121938488.491b73821b1517947c4dd95c17aad62c. 2023-07-12 00:32:19,724 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1689121938488.491b73821b1517947c4dd95c17aad62c. 2023-07-12 00:32:19,724 WARN [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34407] master.MasterRpcServices(612): jenkins-hbase20.apache.org,39949,1689121937155 reported a fatal error: ***** ABORTING region server jenkins-hbase20.apache.org,39949,1689121937155: killAll ***** 2023-07-12 00:32:19,724 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 491b73821b1517947c4dd95c17aad62c: 2023-07-12 00:32:19,725 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1689121938488.491b73821b1517947c4dd95c17aad62c. 2023-07-12 00:32:19,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 4bf803cbe0da319f9582ad75165052ed, disabling compactions & flushes 2023-07-12 00:32:19,726 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1689121938488.4bf803cbe0da319f9582ad75165052ed. 2023-07-12 00:32:19,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1689121938488.4bf803cbe0da319f9582ad75165052ed. 2023-07-12 00:32:19,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1689121938488.4bf803cbe0da319f9582ad75165052ed. after waiting 0 ms 2023-07-12 00:32:19,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1689121938488.4bf803cbe0da319f9582ad75165052ed. 2023-07-12 00:32:19,726 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,39949,1689121937155' ***** 2023-07-12 00:32:19,726 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2311): STOPPED: killAll 2023-07-12 00:32:19,726 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase20.apache.org,41331,1689121938396: killAll ***** 2023-07-12 00:32:19,726 ERROR [Listener at localhost.localdomain/34481] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 00:32:19,726 INFO [RS:1;jenkins-hbase20:39949] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 00:32:19,727 INFO [RS:1;jenkins-hbase20:39949] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-12 00:32:19,727 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 00:32:19,727 INFO [RS:1;jenkins-hbase20:39949] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-12 00:32:19,727 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-12 00:32:19,727 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1689121938488.4bf803cbe0da319f9582ad75165052ed. 2023-07-12 00:32:19,727 INFO [RS:1;jenkins-hbase20:39949] regionserver.HRegionServer(3305): Received CLOSE for ed9da68fb58a11fa382831aaae424ddd 2023-07-12 00:32:19,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 4bf803cbe0da319f9582ad75165052ed: 2023-07-12 00:32:19,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1689121938488.4bf803cbe0da319f9582ad75165052ed. 2023-07-12 00:32:19,728 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-12 00:32:19,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing ed9da68fb58a11fa382831aaae424ddd, disabling compactions & flushes 2023-07-12 00:32:19,728 INFO [RS:1;jenkins-hbase20:39949] regionserver.HRegionServer(3305): Received CLOSE for 6b5039c25691704ddb23db278cb8e6c5 2023-07-12 00:32:19,729 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689121937841.ed9da68fb58a11fa382831aaae424ddd. 2023-07-12 00:32:19,729 INFO [RS:1;jenkins-hbase20:39949] regionserver.HRegionServer(3305): Received CLOSE for 7a3cc4296899653225169eafd8b2a036 2023-07-12 00:32:19,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689121937841.ed9da68fb58a11fa382831aaae424ddd. 2023-07-12 00:32:19,730 INFO [RS:1;jenkins-hbase20:39949] regionserver.HRegionServer(3305): Received CLOSE for b4ca85da7c922b089aeee8eea982fedf 2023-07-12 00:32:19,730 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689121937841.ed9da68fb58a11fa382831aaae424ddd. after waiting 0 ms 2023-07-12 00:32:19,730 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-12 00:32:19,730 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689121937841.ed9da68fb58a11fa382831aaae424ddd. 2023-07-12 00:32:19,730 INFO [RS:1;jenkins-hbase20:39949] regionserver.HRegionServer(3305): Received CLOSE for d91c406970e2b1461f63e6cb82f78d8d 2023-07-12 00:32:19,730 INFO [RS:1;jenkins-hbase20:39949] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,39949,1689121937155 2023-07-12 00:32:19,730 DEBUG [RS:1;jenkins-hbase20:39949] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6d055514 to 127.0.0.1:58965 2023-07-12 00:32:19,730 DEBUG [Listener at localhost.localdomain/34481] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-12 00:32:19,730 DEBUG [RS:1;jenkins-hbase20:39949] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:19,730 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689121937841.ed9da68fb58a11fa382831aaae424ddd. 2023-07-12 00:32:19,730 INFO [RS:1;jenkins-hbase20:39949] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 00:32:19,730 INFO [RS:1;jenkins-hbase20:39949] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 00:32:19,730 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689121937841.ed9da68fb58a11fa382831aaae424ddd. 2023-07-12 00:32:19,730 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for ed9da68fb58a11fa382831aaae424ddd: 2023-07-12 00:32:19,730 INFO [RS:1;jenkins-hbase20:39949] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 00:32:19,730 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689121937841.ed9da68fb58a11fa382831aaae424ddd. 2023-07-12 00:32:19,730 INFO [RS:1;jenkins-hbase20:39949] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-12 00:32:19,731 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 6b5039c25691704ddb23db278cb8e6c5, disabling compactions & flushes 2023-07-12 00:32:19,731 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1689121938488.6b5039c25691704ddb23db278cb8e6c5. 2023-07-12 00:32:19,731 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1689121938488.6b5039c25691704ddb23db278cb8e6c5. 2023-07-12 00:32:19,731 INFO [RS:1;jenkins-hbase20:39949] regionserver.HRegionServer(1474): Waiting on 5 regions to close 2023-07-12 00:32:19,731 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1689121938488.6b5039c25691704ddb23db278cb8e6c5. after waiting 0 ms 2023-07-12 00:32:19,731 DEBUG [RS:1;jenkins-hbase20:39949] regionserver.HRegionServer(1478): Online Regions={6b5039c25691704ddb23db278cb8e6c5=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1689121938488.6b5039c25691704ddb23db278cb8e6c5., 7a3cc4296899653225169eafd8b2a036=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1689121938488.7a3cc4296899653225169eafd8b2a036., 1588230740=hbase:meta,,1.1588230740, b4ca85da7c922b089aeee8eea982fedf=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1689121938488.b4ca85da7c922b089aeee8eea982fedf., d91c406970e2b1461f63e6cb82f78d8d=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1689121938488.d91c406970e2b1461f63e6cb82f78d8d.} 2023-07-12 00:32:19,732 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 00:32:19,732 DEBUG [RS:1;jenkins-hbase20:39949] regionserver.HRegionServer(1504): Waiting on 1588230740, 6b5039c25691704ddb23db278cb8e6c5, 7a3cc4296899653225169eafd8b2a036, b4ca85da7c922b089aeee8eea982fedf, d91c406970e2b1461f63e6cb82f78d8d 2023-07-12 00:32:19,731 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1689121938488.6b5039c25691704ddb23db278cb8e6c5. 2023-07-12 00:32:19,732 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 00:32:19,733 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 00:32:19,733 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 00:32:19,733 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 00:32:19,733 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1192755200, "init": 524288000, "max": 2051014656, "used": 575443152 }, "NonHeapMemoryUsage": { "committed": 144072704, "init": 2555904, "max": -1, "used": 141449392 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 0, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 45102, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 1, "ProcessCallTime_num_ops": 167, "ProcessCallTime_min": 0, "ProcessCallTime_max": 23, "ProcessCallTime_mean": 2, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 1, "ProcessCallTime_75th_percentile": 3, "ProcessCallTime_90th_percentile": 4, "ProcessCallTime_95th_percentile": 9, "ProcessCallTime_98th_percentile": 17, "ProcessCallTime_99th_percentile": 23, "ProcessCallTime_99.9th_percentile": 23, "ProcessCallTime_TimeRangeCount_0-1": 30, "ProcessCallTime_TimeRangeCount_1-3": 12, "ProcessCallTime_TimeRangeCount_3-10": 9, "ProcessCallTime_TimeRangeCount_10-30": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 167, "QueueCallTime_min": 0, "QueueCallTime_max": 6, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 1, "QueueCallTime_95th_percentile": 2, "QueueCallTime_98th_percentile": 3, "QueueCallTime_99th_percentile": 6, "QueueCallTime_99.9th_percentile": 6, "QueueCallTime_TimeRangeCount_0-1": 49, "QueueCallTime_TimeRangeCount_1-3": 3, "QueueCallTime_TimeRangeCount_3-10": 1, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 167, "TotalCallTime_min": 0, "TotalCallTime_max": 23, "TotalCallTime_mean": 2, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 1, "TotalCallTime_75th_percentile": 4, "TotalCallTime_90th_percentile": 7, "TotalCallTime_95th_percentile": 9, "TotalCallTime_98th_percentile": 17, "TotalCallTime_99th_percentile": 23, "TotalCallTime_99.9th_percentile": 23, "TotalCallTime_TimeRangeCount_0-1": 27, "TotalCallTime_TimeRangeCount_1-3": 12, "TotalCallTime_TimeRangeCount_3-10": 12, "TotalCallTime_TimeRangeCount_10-30": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 167, "ResponseSize_min": 0, "ResponseSize_max": 5990, "ResponseSize_mean": 208, "ResponseSize_25th_percentile": 1, "ResponseSize_median": 2, "ResponseSize_75th_percentile": 5, "ResponseSize_90th_percentile": 769, "ResponseSize_95th_percentile": 2683, "ResponseSize_98th_percentile": 4667, "ResponseSize_99th_percentile": 5328, "ResponseSize_99.9th_percentile": 5923, "ResponseSize_SizeRangeCount_0-10": 43, "ResponseSize_SizeRangeCount_10-100": 3, "ResponseSize_SizeRangeCount_100-1000": 7, "exceptions.UnknownScannerException": 0, "exceptions": 2, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 1, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 167, "RequestSize_min": 80, "RequestSize_max": 2899, "RequestSize_mean": 269, "RequestSize_25th_percentile": 145, "RequestSize_median": 176, "RequestSize_75th_percentile": 339, "RequestSize_90th_percentile": 1009, "RequestSize_95th_percentile": 1954, "RequestSize_98th_percentile": 2521, "RequestSize_99th_percentile": 2710, "RequestSize_99.9th_percentile": 2880, "RequestSize_SizeRangeCount_10-100": 5, "RequestSize_SizeRangeCount_100-1000": 48, "sentBytes": 28192 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:58965", "tag.serverName": "jenkins-hbase20.apache.org,42069,1689121928319", "tag.clusterId": "b855a1de-f551-45f5-8b9b-f7b8ff57b75f", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "regionCount": 4, "storeCount": 6, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 2, "maxStoreFileCount": 1, "memStoreSize": 78, "storeFileSize": 21223, "maxStoreFileAge": 385, "minStoreFileAge": 333, "avgStoreFileAge": 359, "numReferenceFiles": 0, "regionServerStartTime": 1689121928319, "averageRegionSize": 5325, "storeFileIndexSize": 920, "staticIndexSize": 290, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 79133, "localBytesRead": 79133, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819794032, "blockCacheCount": 2, "blockCacheDataBlockCount": 2, "blockCacheSize": 611856, "blockCacheCountHitPercent": 50.0, "blockCacheExpressHitPercent": 50.0, "l1CacheSize": 611856, "l1CacheFreeSize": 819794032, "l1CacheCount": 2, "l1CacheEvictionCount": 0, "l1CacheHitCount": 2, "l1CacheMissCount": 2, "l1CacheHitRatio": 0.5, "l1CacheMissRatio": 0.5, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 1.2, "writeRequestRatePerSecond": 0.4, "ByteBuffAllocatorHeapAllocationBytes": 147754, "ByteBuffAllocatorPoolAllocationBytes": 599040, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 12, "totalRowActionRequestCount": 8, "readRequestCount": 6, "filteredReadRequestCount": 0, "writeRequestCount": 2, "rpcGetRequestCount": 7, "rpcFullScanRequestCount": 0, "rpcScanRequestCount": 3, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 2, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 2, "blockCacheHitCountPrimary": 2, "blockCacheHitCachingCount": 2, "blockCacheMissCount": 2, "blockCacheMissCountPrimary": 2, "blockCacheMissCachingCount": 2, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 2, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 2, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 74, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 11438, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 3, "ScanTime_min": 0, "ScanTime_max": 0, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 0, "ScanTime_75th_percentile": 0, "ScanTime_90th_percentile": 0, "ScanTime_95th_percentile": 0, "ScanTime_98th_percentile": 0, "ScanTime_99th_percentile": 0, "ScanTime_99.9th_percentile": 0, "ScanTime_TimeRangeCount_0-1": 3, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 2, "Put_min": 1, "Put_max": 3, "Put_mean": 2, "Put_25th_percentile": 1, "Put_median": 2, "Put_75th_percentile": 3, "Put_90th_percentile": 3, "Put_95th_percentile": 3, "Put_98th_percentile": 3, "Put_99th_percentile": 3, "Put_99.9th_percentile": 3, "Put_TimeRangeCount_0-1": 1, "Put_TimeRangeCount_1-3": 1, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 1, "FlushMemstoreSize_min": 11438, "FlushMemstoreSize_max": 11438, "FlushMemstoreSize_mean": 11438, "FlushMemstoreSize_25th_percentile": 11438, "FlushMemstoreSize_median": 11438, "FlushMemstoreSize_75th_percentile": 11438, "FlushMemstoreSize_90th_percentile": 11438, "FlushMemstoreSize_95th_percentile": 11438, "FlushMemstoreSize_98th_percentile": 11438, "FlushMemstoreSize_99th_percentile": 11438, "FlushMemstoreSize_99.9th_percentile": 11438, "FlushMemstoreSize_SizeRangeCount_100-1000": 1, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 6, "Get_min": 0, "Get_max": 23, "Get_mean": 4, "Get_25th_percentile": 0, "Get_median": 1, "Get_75th_percentile": 2, "Get_90th_percentile": 23, "Get_95th_percentile": 23, "Get_98th_percentile": 23, "Get_99th_percentile": 23, "Get_99.9th_percentile": 23, "Get_TimeRangeCount_0-1": 4, "Get_TimeRangeCount_1-3": 1, "Get_TimeRangeCount_10-30": 1, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 21223, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 1, "FlushTime_min": 154, "FlushTime_max": 154, "FlushTime_mean": 154, "FlushTime_25th_percentile": 154, "FlushTime_median": 154, "FlushTime_75th_percentile": 154, "FlushTime_90th_percentile": 154, "FlushTime_95th_percentile": 154, "FlushTime_98th_percentile": 154, "FlushTime_99th_percentile": 154, "FlushTime_99.9th_percentile": 154, "FlushTime_TimeRangeCount_100-300": 1, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 3, "ScanSize_min": 86, "ScanSize_max": 985, "ScanSize_mean": 685, "ScanSize_25th_percentile": 86, "ScanSize_median": 438, "ScanSize_75th_percentile": 711, "ScanSize_90th_percentile": 875, "ScanSize_95th_percentile": 930, "ScanSize_98th_percentile": 963, "ScanSize_99th_percentile": 974, "ScanSize_99.9th_percentile": 983, "ScanSize_SizeRangeCount_10-100": 1, "ScanSize_SizeRangeCount_100-1000": 2, "slowGetCount": 0, "flushedMemstoreBytes": 11438, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 1, "FlushOutputSize_min": 21223, "FlushOutputSize_max": 21223, "FlushOutputSize_mean": 21223, "FlushOutputSize_25th_percentile": 21223, "FlushOutputSize_median": 21223, "FlushOutputSize_75th_percentile": 21223, "FlushOutputSize_90th_percentile": 21223, "FlushOutputSize_95th_percentile": 21223, "FlushOutputSize_98th_percentile": 21223, "FlushOutputSize_99th_percentile": 21223, "FlushOutputSize_99.9th_percentile": 21223, "FlushOutputSize_SizeRangeCount_100-1000": 1, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 93, "ServerReadQueryPerSecond_mean_rate": 3.0241271332370667, "ServerReadQueryPerSecond_1min_rate": 2.81760632426473, "ServerReadQueryPerSecond_5min_rate": 2.6383756459467356, "ServerReadQueryPerSecond_15min_rate": 2.6124191975460884, "ServerWriteQueryPerSecond_count": 74, "ServerWriteQueryPerSecond_mean_rate": 2.406294348075967, "ServerWriteQueryPerSecond_1min_rate": 1.902961977749651, "ServerWriteQueryPerSecond_5min_rate": 1.5054020168326836, "ServerWriteQueryPerSecond_15min_rate": 1.4354140700006486 } ] } 2023-07-12 00:32:19,734 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34407] master.MasterRpcServices(612): jenkins-hbase20.apache.org,41331,1689121938396 reported a fatal error: ***** ABORTING region server jenkins-hbase20.apache.org,41331,1689121938396: killAll ***** 2023-07-12 00:32:19,734 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1689121938488.6b5039c25691704ddb23db278cb8e6c5. 2023-07-12 00:32:19,734 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 6b5039c25691704ddb23db278cb8e6c5: 2023-07-12 00:32:19,734 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1689121938488.6b5039c25691704ddb23db278cb8e6c5. 2023-07-12 00:32:19,734 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1825): Memstore data size is 20353 in region hbase:meta,,1.1588230740 2023-07-12 00:32:19,734 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,41331,1689121938396' ***** 2023-07-12 00:32:19,735 INFO [Listener at localhost.localdomain/34481] regionserver.HRegionServer(2311): STOPPED: killAll 2023-07-12 00:32:19,735 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 00:32:19,735 INFO [RS:2;jenkins-hbase20:41331] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 00:32:19,736 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 00:32:19,736 INFO [RS:2;jenkins-hbase20:41331] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-12 00:32:19,736 INFO [RS:2;jenkins-hbase20:41331] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-12 00:32:19,736 INFO [RS:2;jenkins-hbase20:41331] regionserver.HRegionServer(3305): Received CLOSE for 591a2fafd71a4502f0c9d7c65c6de8cd 2023-07-12 00:32:19,736 INFO [RS:2;jenkins-hbase20:41331] regionserver.HRegionServer(3305): Received CLOSE for 02bb0fac148ca555b93632ec43604318 2023-07-12 00:32:19,736 INFO [RS:2;jenkins-hbase20:41331] regionserver.HRegionServer(3305): Received CLOSE for 335cd564cec52505cf0fa421b43d2650 2023-07-12 00:32:19,735 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 7a3cc4296899653225169eafd8b2a036, disabling compactions & flushes 2023-07-12 00:32:19,737 INFO [RS:2;jenkins-hbase20:41331] regionserver.HRegionServer(3305): Received CLOSE for b36a989038618469dbf3c92ec0b243a1 2023-07-12 00:32:19,737 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1689121938488.7a3cc4296899653225169eafd8b2a036. 2023-07-12 00:32:19,737 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 591a2fafd71a4502f0c9d7c65c6de8cd, disabling compactions & flushes 2023-07-12 00:32:19,736 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 00:32:19,736 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 00:32:19,735 INFO [Listener at localhost.localdomain/34481] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-12 00:32:19,737 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-12 00:32:19,737 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1689121938488.591a2fafd71a4502f0c9d7c65c6de8cd. 2023-07-12 00:32:19,737 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1689121938488.7a3cc4296899653225169eafd8b2a036. 2023-07-12 00:32:19,737 INFO [RS:2;jenkins-hbase20:41331] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,41331,1689121938396 2023-07-12 00:32:19,738 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1689121938488.7a3cc4296899653225169eafd8b2a036. after waiting 0 ms 2023-07-12 00:32:19,738 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1689121938488.591a2fafd71a4502f0c9d7c65c6de8cd. 2023-07-12 00:32:19,738 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f696882 to 127.0.0.1:58965 2023-07-12 00:32:19,739 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1689121938488.591a2fafd71a4502f0c9d7c65c6de8cd. after waiting 0 ms 2023-07-12 00:32:19,739 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1689121938488.591a2fafd71a4502f0c9d7c65c6de8cd. 2023-07-12 00:32:19,739 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1689121938488.7a3cc4296899653225169eafd8b2a036. 2023-07-12 00:32:19,739 DEBUG [RS:2;jenkins-hbase20:41331] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x15ad5174 to 127.0.0.1:58965 2023-07-12 00:32:19,739 DEBUG [Listener at localhost.localdomain/34481] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:19,739 DEBUG [RS:2;jenkins-hbase20:41331] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:19,739 DEBUG [Listener at localhost.localdomain/34481] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-12 00:32:19,739 INFO [RS:2;jenkins-hbase20:41331] regionserver.HRegionServer(1474): Waiting on 4 regions to close 2023-07-12 00:32:19,739 DEBUG [RS:2;jenkins-hbase20:41331] regionserver.HRegionServer(1478): Online Regions={591a2fafd71a4502f0c9d7c65c6de8cd=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1689121938488.591a2fafd71a4502f0c9d7c65c6de8cd., 02bb0fac148ca555b93632ec43604318=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1689121938488.02bb0fac148ca555b93632ec43604318., 335cd564cec52505cf0fa421b43d2650=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1689121938488.335cd564cec52505cf0fa421b43d2650., b36a989038618469dbf3c92ec0b243a1=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1689121938488.b36a989038618469dbf3c92ec0b243a1.} 2023-07-12 00:32:19,739 DEBUG [RS:2;jenkins-hbase20:41331] regionserver.HRegionServer(1504): Waiting on 02bb0fac148ca555b93632ec43604318, 335cd564cec52505cf0fa421b43d2650, 591a2fafd71a4502f0c9d7c65c6de8cd, b36a989038618469dbf3c92ec0b243a1 2023-07-12 00:32:19,742 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1689121938488.591a2fafd71a4502f0c9d7c65c6de8cd. 2023-07-12 00:32:19,743 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 591a2fafd71a4502f0c9d7c65c6de8cd: 2023-07-12 00:32:19,743 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1689121938488.591a2fafd71a4502f0c9d7c65c6de8cd. 2023-07-12 00:32:19,744 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 02bb0fac148ca555b93632ec43604318, disabling compactions & flushes 2023-07-12 00:32:19,744 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1689121938488.02bb0fac148ca555b93632ec43604318. 2023-07-12 00:32:19,744 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1689121938488.02bb0fac148ca555b93632ec43604318. 2023-07-12 00:32:19,744 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1689121938488.02bb0fac148ca555b93632ec43604318. after waiting 0 ms 2023-07-12 00:32:19,744 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1689121938488.02bb0fac148ca555b93632ec43604318. 2023-07-12 00:32:19,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1689121938488.7a3cc4296899653225169eafd8b2a036. 2023-07-12 00:32:19,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 7a3cc4296899653225169eafd8b2a036: 2023-07-12 00:32:19,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1689121938488.7a3cc4296899653225169eafd8b2a036. 2023-07-12 00:32:19,755 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing b4ca85da7c922b089aeee8eea982fedf, disabling compactions & flushes 2023-07-12 00:32:19,755 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1689121938488.b4ca85da7c922b089aeee8eea982fedf. 2023-07-12 00:32:19,755 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1689121938488.02bb0fac148ca555b93632ec43604318. 2023-07-12 00:32:19,755 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1689121938488.b4ca85da7c922b089aeee8eea982fedf. 2023-07-12 00:32:19,755 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 02bb0fac148ca555b93632ec43604318: 2023-07-12 00:32:19,755 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1689121938488.b4ca85da7c922b089aeee8eea982fedf. after waiting 0 ms 2023-07-12 00:32:19,755 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1689121938488.b4ca85da7c922b089aeee8eea982fedf. 2023-07-12 00:32:19,755 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1689121938488.02bb0fac148ca555b93632ec43604318. 2023-07-12 00:32:19,759 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 335cd564cec52505cf0fa421b43d2650, disabling compactions & flushes 2023-07-12 00:32:19,760 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1689121938488.335cd564cec52505cf0fa421b43d2650. 2023-07-12 00:32:19,760 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1689121938488.335cd564cec52505cf0fa421b43d2650. 2023-07-12 00:32:19,760 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1689121938488.335cd564cec52505cf0fa421b43d2650. after waiting 0 ms 2023-07-12 00:32:19,760 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1689121938488.b4ca85da7c922b089aeee8eea982fedf. 2023-07-12 00:32:19,760 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1689121938488.335cd564cec52505cf0fa421b43d2650. 2023-07-12 00:32:19,760 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for b4ca85da7c922b089aeee8eea982fedf: 2023-07-12 00:32:19,760 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1689121938488.b4ca85da7c922b089aeee8eea982fedf. 2023-07-12 00:32:19,762 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing d91c406970e2b1461f63e6cb82f78d8d, disabling compactions & flushes 2023-07-12 00:32:19,762 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 00:32:19,762 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1689121938488.d91c406970e2b1461f63e6cb82f78d8d. 2023-07-12 00:32:19,763 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 00:32:19,763 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1689121938488.d91c406970e2b1461f63e6cb82f78d8d. 2023-07-12 00:32:19,763 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1689121938488.d91c406970e2b1461f63e6cb82f78d8d. after waiting 0 ms 2023-07-12 00:32:19,763 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1689121938488.d91c406970e2b1461f63e6cb82f78d8d. 2023-07-12 00:32:19,767 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1689121938488.335cd564cec52505cf0fa421b43d2650. 2023-07-12 00:32:19,767 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 335cd564cec52505cf0fa421b43d2650: 2023-07-12 00:32:19,767 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1689121938488.335cd564cec52505cf0fa421b43d2650. 2023-07-12 00:32:19,768 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing b36a989038618469dbf3c92ec0b243a1, disabling compactions & flushes 2023-07-12 00:32:19,768 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1689121938488.b36a989038618469dbf3c92ec0b243a1. 2023-07-12 00:32:19,768 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1689121938488.d91c406970e2b1461f63e6cb82f78d8d. 2023-07-12 00:32:19,768 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1689121938488.b36a989038618469dbf3c92ec0b243a1. 2023-07-12 00:32:19,768 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for d91c406970e2b1461f63e6cb82f78d8d: 2023-07-12 00:32:19,768 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1689121938488.b36a989038618469dbf3c92ec0b243a1. after waiting 0 ms 2023-07-12 00:32:19,768 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1689121938488.b36a989038618469dbf3c92ec0b243a1. 2023-07-12 00:32:19,768 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1689121938488.d91c406970e2b1461f63e6cb82f78d8d. 2023-07-12 00:32:19,769 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1689121938488.b36a989038618469dbf3c92ec0b243a1. 2023-07-12 00:32:19,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for b36a989038618469dbf3c92ec0b243a1: 2023-07-12 00:32:19,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1689121938488.b36a989038618469dbf3c92ec0b243a1. 2023-07-12 00:32:19,770 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 00:32:19,812 WARN [M:1;jenkins-hbase20:34407] zookeeper.ZKUtil(402): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Unable to get data of znode /hbase/master org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:134) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:2131) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:330) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:394) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:145) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:324) at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1527) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1205) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:541) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:32:19,812 ERROR [M:1;jenkins-hbase20:34407] zookeeper.ZKWatcher(701): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:134) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:2131) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:330) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:394) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:145) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:324) at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1527) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1205) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:541) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:32:19,812 DEBUG [M:1;jenkins-hbase20:34407] master.ActiveMasterManager(335): master:34407-0x1007e974181002b, quorum=127.0.0.1:58965, baseZNode=/hbase Failed delete of our master address node; KeeperErrorCode = Session expired for /hbase/master 2023-07-12 00:32:19,812 INFO [M:1;jenkins-hbase20:34407] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 00:32:19,813 INFO [M:1;jenkins-hbase20:34407] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=true 2023-07-12 00:32:19,816 DEBUG [M:0;jenkins-hbase20:45037] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,45037,1689121937044 already deleted, retry=false 2023-07-12 00:32:19,818 DEBUG [M:1;jenkins-hbase20:34407] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 00:32:19,818 INFO [M:1;jenkins-hbase20:34407] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:19,818 DEBUG [M:1;jenkins-hbase20:34407] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:19,818 DEBUG [M:1;jenkins-hbase20:34407] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 00:32:19,818 DEBUG [M:1;jenkins-hbase20:34407] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:19,818 ERROR [M:1;jenkins-hbase20:34407] regionserver.HRegion(1825): Memstore data size is 113865 in region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:19,818 INFO [M:1;jenkins-hbase20:34407] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 00:32:19,818 DEBUG [M:1;jenkins-hbase20:34407] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 00:32:19,818 INFO [M:1;jenkins-hbase20:34407] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 00:32:19,818 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 00:32:19,819 INFO [M:1;jenkins-hbase20:34407] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:34407 2023-07-12 00:32:19,915 WARN [M:1;jenkins-hbase20:34407] regionserver.HRegionServer(1217): Failed deleting my ephemeral node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/jenkins-hbase20.apache.org,34407,1689121937069 at org.apache.zookeeper.KeeperException.create(KeeperException.java:134) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:1793) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:162) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:902) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:894) at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1694) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1213) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:541) at java.lang.Thread.run(Thread.java:750) 2023-07-12 00:32:19,917 INFO [M:1;jenkins-hbase20:34407] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,34407,1689121937069; zookeeper connection closed. 2023-07-12 00:32:19,921 INFO [RS:0;jenkins-hbase20:33511] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,33511,1689121937117; all regions closed. 2023-07-12 00:32:19,930 DEBUG [RS:0;jenkins-hbase20:33511] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:19,930 INFO [RS:0;jenkins-hbase20:33511] regionserver.LeaseManager(133): Closed leases 2023-07-12 00:32:19,931 INFO [RS:0;jenkins-hbase20:33511] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-12 00:32:19,931 INFO [RS:0;jenkins-hbase20:33511] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 00:32:19,931 INFO [RS:0;jenkins-hbase20:33511] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 00:32:19,931 INFO [RS:0;jenkins-hbase20:33511] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 00:32:19,935 INFO [RS:0;jenkins-hbase20:33511] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:33511 2023-07-12 00:32:19,935 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 00:32:19,935 INFO [RS:1;jenkins-hbase20:39949] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,39949,1689121937155; all regions closed. 2023-07-12 00:32:19,940 INFO [RS:2;jenkins-hbase20:41331] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,41331,1689121938396; all regions closed. 2023-07-12 00:32:19,942 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,39949,1689121937155/jenkins-hbase20.apache.org%2C39949%2C1689121937155.1689121937691 not finished, retry = 0 2023-07-12 00:32:19,943 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/0da48482-e5ed-d797-2265-1836e88bd983/WALs/jenkins-hbase20.apache.org,41331,1689121938396/jenkins-hbase20.apache.org%2C41331%2C1689121938396.1689121938587 not finished, retry = 0 2023-07-12 00:32:20,018 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:41331-0x1007e9741810032, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,33511,1689121937117 2023-07-12 00:32:20,018 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:41331-0x1007e9741810032, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:20,018 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:33511-0x1007e974181002c, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,33511,1689121937117 2023-07-12 00:32:20,018 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:39949-0x1007e974181002d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,33511,1689121937117 2023-07-12 00:32:20,018 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:33511-0x1007e974181002c, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:20,018 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:39949-0x1007e974181002d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:20,043 DEBUG [RS:1;jenkins-hbase20:39949] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:20,043 INFO [RS:1;jenkins-hbase20:39949] regionserver.LeaseManager(133): Closed leases 2023-07-12 00:32:20,043 INFO [RS:1;jenkins-hbase20:39949] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-12 00:32:20,043 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 00:32:20,044 INFO [RS:1;jenkins-hbase20:39949] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:39949 2023-07-12 00:32:20,044 DEBUG [RS:2;jenkins-hbase20:41331] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 00:32:20,044 INFO [RS:2;jenkins-hbase20:41331] regionserver.LeaseManager(133): Closed leases 2023-07-12 00:32:20,044 INFO [RS:2;jenkins-hbase20:41331] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-12 00:32:20,044 INFO [RS:2;jenkins-hbase20:41331] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 00:32:20,044 INFO [RS:2;jenkins-hbase20:41331] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 00:32:20,044 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:39949-0x1007e974181002d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,39949,1689121937155 2023-07-12 00:32:20,044 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 00:32:20,044 INFO [RS:2;jenkins-hbase20:41331] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 00:32:20,044 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:41331-0x1007e9741810032, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,39949,1689121937155 2023-07-12 00:32:20,045 INFO [RS:2;jenkins-hbase20:41331] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:41331 2023-07-12 00:32:20,045 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:41331-0x1007e9741810032, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,41331,1689121938396 2023-07-12 00:32:20,116 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:20,116 INFO [M:0;jenkins-hbase20:45037] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,45037,1689121937044; zookeeper connection closed. 2023-07-12 00:32:20,116 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): master:45037-0x1007e974181002a, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:20,216 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:41331-0x1007e9741810032, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:20,216 INFO [RS:2;jenkins-hbase20:41331] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,41331,1689121938396; zookeeper connection closed. 2023-07-12 00:32:20,217 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:41331-0x1007e9741810032, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:20,217 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@4bb0479] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@4bb0479 2023-07-12 00:32:20,317 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:39949-0x1007e974181002d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:20,317 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:39949-0x1007e974181002d, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:20,317 INFO [RS:1;jenkins-hbase20:39949] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,39949,1689121937155; zookeeper connection closed. 2023-07-12 00:32:20,319 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@714cd3d9] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@714cd3d9 2023-07-12 00:32:20,417 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:33511-0x1007e974181002c, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:20,417 INFO [RS:0;jenkins-hbase20:33511] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,33511,1689121937117; zookeeper connection closed. 2023-07-12 00:32:20,417 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): regionserver:33511-0x1007e974181002c, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:20,430 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@284fc1fd] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@284fc1fd 2023-07-12 00:32:20,431 INFO [Listener at localhost.localdomain/34481] util.JVMClusterUtil(335): Shutdown of 2 master(s) and 3 regionserver(s) complete 2023-07-12 00:32:20,532 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810029, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:20,533 INFO [Listener at localhost.localdomain/34481] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 00:32:20,538 INFO [Listener at localhost.localdomain/34481] zookeeper.RecoverableZooKeeper(93): Process identifier=testing utility connecting to ZooKeeper ensemble=127.0.0.1:58965 2023-07-12 00:32:20,548 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility0x0, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 00:32:20,548 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility0x0, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2023-07-12 00:32:20,549 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility0x0, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/rs 2023-07-12 00:32:20,550 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility0x0, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/splitWAL 2023-07-12 00:32:20,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): testing utility-0x1007e9741810035 connected 2023-07-12 00:32:20,552 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters 2023-07-12 00:32:20,553 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc 2023-07-12 00:32:20,553 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/master-maintenance 2023-07-12 00:32:20,554 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot 2023-07-12 00:32:20,554 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/switch 2023-07-12 00:32:20,554 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-12 00:32:20,555 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/draining 2023-07-12 00:32:20,556 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-07-12 00:32:20,556 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/hbaseid 2023-07-12 00:32:20,557 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/table 2023-07-12 00:32:20,565 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/abort 2023-07-12 00:32:20,565 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired 2023-07-12 00:32:20,566 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached 2023-07-12 00:32:20,568 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/abort 2023-07-12 00:32:20,569 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired 2023-07-12 00:32:20,570 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached 2023-07-12 00:32:20,572 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/default 2023-07-12 00:32:20,573 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/hbase 2023-07-12 00:32:20,574 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/table/hbase:meta 2023-07-12 00:32:20,575 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/table/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2023-07-12 00:32:20,575 DEBUG [Listener at localhost.localdomain/34481] zookeeper.ZKUtil(162): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Set watcher on existing znode=/hbase/table/hbase:namespace 2023-07-12 00:32:20,586 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table/hbase:namespace 2023-07-12 00:32:20,587 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/table 2023-07-12 00:32:20,587 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2023-07-12 00:32:20,588 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table/hbase:meta 2023-07-12 00:32:20,588 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 00:32:20,588 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 00:32:20,588 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 00:32:20,588 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached 2023-07-12 00:32:20,588 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot 2023-07-12 00:32:20,588 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 00:32:20,588 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 00:32:20,588 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached 2023-07-12 00:32:20,588 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc 2023-07-12 00:32:20,588 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 00:32:20,588 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 00:32:20,588 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table 2023-07-12 00:32:20,588 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 00:32:20,588 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/hbaseid 2023-07-12 00:32:20,588 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace 2023-07-12 00:32:20,588 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/draining 2023-07-12 00:32:20,588 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 00:32:20,588 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/switch 2023-07-12 00:32:20,588 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot 2023-07-12 00:32:20,588 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master-maintenance 2023-07-12 00:32:20,588 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc 2023-07-12 00:32:20,588 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 00:32:20,588 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL 2023-07-12 00:32:20,588 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs 2023-07-12 00:32:20,588 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 00:32:20,589 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase 2023-07-12 00:32:20,634 INFO [Listener at localhost.localdomain/34481] hbase.ResourceChecker(175): after: TestZooKeeper#testRegionAssignmentAfterMasterRecoveryDueToZKExpiry Thread=401 (was 373) Potentially hanging thread: RS-EventLoopGroup-23-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-24-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-15 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1879248547) connection to localhost.localdomain/127.0.0.1:44295 from jenkins.hfs.11 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-14 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: master/jenkins-hbase20:0:becomeActiveMaster java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:118) org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:47) org.apache.hadoop.hbase.master.region.MasterRegion.replayWALs(MasterRegion.java:295) org.apache.hadoop.hbase.master.region.MasterRegion.open(MasterRegion.java:255) org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:344) org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) org.apache.hadoop.hbase.master.HMaster$$Lambda$44/337926966.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-20-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1879248547) connection to localhost.localdomain/127.0.0.1:44295 from jenkins.hfs.10 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-22-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-24-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-22-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-21-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1879248547) connection to localhost.localdomain/127.0.0.1:44295 from jenkins.hfs.9 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: LeaseRenewer:jenkins.hfs.10@localhost.localdomain:44295 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1628992759_17 at /127.0.0.1:37226 [Waiting for operation #15] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-21-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-21-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-24-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: master:store-WAL-Roller java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.wal.AbstractWALRoller.run(AbstractWALRoller.java:179) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1628992759_17 at /127.0.0.1:37244 [Waiting for operation #7] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-23-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1628992759_17 at /127.0.0.1:33290 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-23-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1628992759_17 at /127.0.0.1:47422 [Waiting for operation #36] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.11@localhost.localdomain:44295 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.9@localhost.localdomain:44295 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=807 (was 759) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=216 (was 216), ProcessCount=173 (was 173), AvailableMemoryMB=4834 (was 4878) 2023-07-12 00:32:20,636 INFO [Listener at localhost.localdomain/34481] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-12 00:32:20,738 DEBUG [Listener at localhost.localdomain/34481-EventThread] zookeeper.ZKWatcher(600): testing utility-0x1007e9741810035, quorum=127.0.0.1:58965, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 00:32:20,739 WARN [Listener at localhost.localdomain/34481] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-12 00:32:20,746 INFO [Listener at localhost.localdomain/34481] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 00:32:20,851 WARN [BP-1731987413-148.251.75.209-1689121899843 heartbeating to localhost.localdomain/127.0.0.1:44295] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-12 00:32:20,851 WARN [BP-1731987413-148.251.75.209-1689121899843 heartbeating to localhost.localdomain/127.0.0.1:44295] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1731987413-148.251.75.209-1689121899843 (Datanode Uuid 6fb7e7dc-4325-40ec-b46c-1693148a1c05) service to localhost.localdomain/127.0.0.1:44295 2023-07-12 00:32:20,854 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/cluster_2bc97589-c177-c63b-7f92-2672baa4b2fe/dfs/data/data3/current/BP-1731987413-148.251.75.209-1689121899843] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 00:32:20,854 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/cluster_2bc97589-c177-c63b-7f92-2672baa4b2fe/dfs/data/data4/current/BP-1731987413-148.251.75.209-1689121899843] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 00:32:20,855 WARN [Listener at localhost.localdomain/34481] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-12 00:32:20,860 INFO [Listener at localhost.localdomain/34481] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 00:32:20,865 WARN [BP-1731987413-148.251.75.209-1689121899843 heartbeating to localhost.localdomain/127.0.0.1:44295] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-12 00:32:20,865 WARN [BP-1731987413-148.251.75.209-1689121899843 heartbeating to localhost.localdomain/127.0.0.1:44295] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1731987413-148.251.75.209-1689121899843 (Datanode Uuid 55ed1f65-bd9a-4ab5-8a8a-ac56d85c09c2) service to localhost.localdomain/127.0.0.1:44295 2023-07-12 00:32:20,866 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/cluster_2bc97589-c177-c63b-7f92-2672baa4b2fe/dfs/data/data1/current/BP-1731987413-148.251.75.209-1689121899843] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 00:32:20,867 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5bddf06-3214-132e-3aff-0cb0e18b8c09/cluster_2bc97589-c177-c63b-7f92-2672baa4b2fe/dfs/data/data2/current/BP-1731987413-148.251.75.209-1689121899843] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 00:32:20,929 INFO [Listener at localhost.localdomain/34481] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-07-12 00:32:20,953 INFO [Listener at localhost.localdomain/34481] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-12 00:32:21,098 INFO [Listener at localhost.localdomain/34481] hbase.HBaseTestingUtility(1293): Minicluster is down