2023-02-13 17:53:31,984 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.TestRetainAssignmentOnRestart timeout: 13 mins 2023-02-13 17:53:31,987 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.TestRetainAssignmentOnRestartSplitWithoutZk timeout: 13 mins 2023-02-13 17:53:32,027 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.TestRetainAssignmentOnRestartSplitWithoutZk#testRetainAssignmentOnSingleRSRestart Thread=7, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=389, ProcessCount=168, AvailableMemoryMB=2840 2023-02-13 17:53:32,295 DEBUG [Time-limited test] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06 2023-02-13 17:53:32,297 INFO [Time-limited test] master.AbstractTestRestartCluster(45): WAL splitting coordinated by zk false 2023-02-13 17:53:32,300 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-13 17:53:32,301 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/cluster_76469730-fa10-8e70-bc1e-b5e0ec8a135a, deleteOnExit=true 2023-02-13 17:53:32,301 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-13 17:53:32,301 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/test.cache.data in system properties and HBase conf 2023-02-13 17:53:32,302 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/hadoop.tmp.dir in system properties and HBase conf 2023-02-13 17:53:32,302 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/hadoop.log.dir in system properties and HBase conf 2023-02-13 17:53:32,303 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-13 17:53:32,303 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-13 17:53:32,303 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-13 17:53:32,408 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-02-13 17:53:32,822 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-13 17:53:32,828 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-13 17:53:32,828 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-13 17:53:32,829 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-13 17:53:32,829 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-13 17:53:32,830 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-13 17:53:32,830 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-13 17:53:32,831 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-13 17:53:32,831 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-13 17:53:32,832 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-13 17:53:32,832 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/nfs.dump.dir in system properties and HBase conf 2023-02-13 17:53:32,832 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/java.io.tmpdir in system properties and HBase conf 2023-02-13 17:53:32,833 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-13 17:53:32,833 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-13 17:53:32,834 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-13 17:53:33,391 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-13 17:53:33,396 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-13 17:53:33,698 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-02-13 17:53:33,863 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-02-13 17:53:33,894 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 17:53:33,937 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 17:53:33,970 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/java.io.tmpdir/Jetty_localhost_45035_hdfs____.wy825h/webapp 2023-02-13 17:53:34,115 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45035 2023-02-13 17:53:34,134 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-13 17:53:34,134 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-13 17:53:34,706 WARN [Listener at localhost/42429] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 17:53:34,788 WARN [Listener at localhost/42429] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 17:53:34,812 WARN [Listener at localhost/42429] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 17:53:34,820 INFO [Listener at localhost/42429] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 17:53:34,827 INFO [Listener at localhost/42429] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/java.io.tmpdir/Jetty_localhost_33093_datanode____homkep/webapp 2023-02-13 17:53:34,958 INFO [Listener at localhost/42429] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33093 2023-02-13 17:53:35,400 WARN [Listener at localhost/40187] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 17:53:35,422 WARN [Listener at localhost/40187] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 17:53:35,429 WARN [Listener at localhost/40187] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 17:53:35,432 INFO [Listener at localhost/40187] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 17:53:35,438 INFO [Listener at localhost/40187] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/java.io.tmpdir/Jetty_localhost_39695_datanode____.z5djl9/webapp 2023-02-13 17:53:35,572 INFO [Listener at localhost/40187] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39695 2023-02-13 17:53:35,592 WARN [Listener at localhost/41871] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 17:53:35,635 WARN [Listener at localhost/41871] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 17:53:35,639 WARN [Listener at localhost/41871] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 17:53:35,642 INFO [Listener at localhost/41871] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 17:53:35,660 INFO [Listener at localhost/41871] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/java.io.tmpdir/Jetty_localhost_43117_datanode____.tl2x9d/webapp 2023-02-13 17:53:35,788 INFO [Listener at localhost/41871] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43117 2023-02-13 17:53:35,816 WARN [Listener at localhost/43465] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 17:53:36,005 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xa603d45c2c59abfe: Processing first storage report for DS-07b12a43-1f57-44a8-a528-3a10ed60ab98 from datanode ac9475ad-04ff-47e5-82bc-722ccbf8b5e7 2023-02-13 17:53:36,007 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xa603d45c2c59abfe: from storage DS-07b12a43-1f57-44a8-a528-3a10ed60ab98 node DatanodeRegistration(127.0.0.1:33739, datanodeUuid=ac9475ad-04ff-47e5-82bc-722ccbf8b5e7, infoPort=33155, infoSecurePort=0, ipcPort=40187, storageInfo=lv=-57;cid=testClusterID;nsid=959185906;c=1676310813473), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-02-13 17:53:36,007 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xe71ef4a049c45adc: Processing first storage report for DS-6ebf9491-78fa-49e0-8c5b-82ee1005d4af from datanode c7394c73-8463-404a-9d61-e450c05adbdc 2023-02-13 17:53:36,007 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xe71ef4a049c45adc: from storage DS-6ebf9491-78fa-49e0-8c5b-82ee1005d4af node DatanodeRegistration(127.0.0.1:33275, datanodeUuid=c7394c73-8463-404a-9d61-e450c05adbdc, infoPort=44309, infoSecurePort=0, ipcPort=43465, storageInfo=lv=-57;cid=testClusterID;nsid=959185906;c=1676310813473), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 17:53:36,007 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x5c67b29cac275754: Processing first storage report for DS-0de4ecf6-a9b9-4f55-81ee-f85a32d928d3 from datanode ce050643-8f71-4f57-ad61-72ab186e54bd 2023-02-13 17:53:36,007 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x5c67b29cac275754: from storage DS-0de4ecf6-a9b9-4f55-81ee-f85a32d928d3 node DatanodeRegistration(127.0.0.1:45833, datanodeUuid=ce050643-8f71-4f57-ad61-72ab186e54bd, infoPort=36777, infoSecurePort=0, ipcPort=41871, storageInfo=lv=-57;cid=testClusterID;nsid=959185906;c=1676310813473), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 17:53:36,007 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xa603d45c2c59abfe: Processing first storage report for DS-a5d0a4c3-a328-475e-b420-397fb7cd201c from datanode ac9475ad-04ff-47e5-82bc-722ccbf8b5e7 2023-02-13 17:53:36,007 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xa603d45c2c59abfe: from storage DS-a5d0a4c3-a328-475e-b420-397fb7cd201c node DatanodeRegistration(127.0.0.1:33739, datanodeUuid=ac9475ad-04ff-47e5-82bc-722ccbf8b5e7, infoPort=33155, infoSecurePort=0, ipcPort=40187, storageInfo=lv=-57;cid=testClusterID;nsid=959185906;c=1676310813473), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 17:53:36,007 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xe71ef4a049c45adc: Processing first storage report for DS-fe112ea1-29ce-41d5-ade3-4173ac825954 from datanode c7394c73-8463-404a-9d61-e450c05adbdc 2023-02-13 17:53:36,008 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xe71ef4a049c45adc: from storage DS-fe112ea1-29ce-41d5-ade3-4173ac825954 node DatanodeRegistration(127.0.0.1:33275, datanodeUuid=c7394c73-8463-404a-9d61-e450c05adbdc, infoPort=44309, infoSecurePort=0, ipcPort=43465, storageInfo=lv=-57;cid=testClusterID;nsid=959185906;c=1676310813473), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-13 17:53:36,008 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x5c67b29cac275754: Processing first storage report for DS-6df7e556-9328-4de5-9cf4-239f67715d09 from datanode ce050643-8f71-4f57-ad61-72ab186e54bd 2023-02-13 17:53:36,008 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x5c67b29cac275754: from storage DS-6df7e556-9328-4de5-9cf4-239f67715d09 node DatanodeRegistration(127.0.0.1:45833, datanodeUuid=ce050643-8f71-4f57-ad61-72ab186e54bd, infoPort=36777, infoSecurePort=0, ipcPort=41871, storageInfo=lv=-57;cid=testClusterID;nsid=959185906;c=1676310813473), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 17:53:36,248 DEBUG [Listener at localhost/43465] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06 2023-02-13 17:53:36,311 INFO [Listener at localhost/43465] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/cluster_76469730-fa10-8e70-bc1e-b5e0ec8a135a/zookeeper_0, clientPort=52890, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/cluster_76469730-fa10-8e70-bc1e-b5e0ec8a135a/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/cluster_76469730-fa10-8e70-bc1e-b5e0ec8a135a/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-13 17:53:36,324 INFO [Listener at localhost/43465] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=52890 2023-02-13 17:53:36,337 INFO [Listener at localhost/43465] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:36,340 INFO [Listener at localhost/43465] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:37,030 INFO [Listener at localhost/43465] util.FSUtils(479): Created version file at hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2 with version=8 2023-02-13 17:53:37,031 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/hbase-staging 2023-02-13 17:53:37,353 INFO [Listener at localhost/43465] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-13 17:53:37,899 INFO [Listener at localhost/43465] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 17:53:37,935 INFO [Listener at localhost/43465] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:37,936 INFO [Listener at localhost/43465] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:37,936 INFO [Listener at localhost/43465] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 17:53:37,936 INFO [Listener at localhost/43465] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:37,936 INFO [Listener at localhost/43465] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 17:53:38,084 INFO [Listener at localhost/43465] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 17:53:38,159 DEBUG [Listener at localhost/43465] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-02-13 17:53:38,257 INFO [Listener at localhost/43465] ipc.NettyRpcServer(120): Bind to /172.31.14.131:38985 2023-02-13 17:53:38,267 INFO [Listener at localhost/43465] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:38,269 INFO [Listener at localhost/43465] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:38,290 INFO [Listener at localhost/43465] zookeeper.RecoverableZooKeeper(93): Process identifier=master:38985 connecting to ZooKeeper ensemble=127.0.0.1:52890 2023-02-13 17:53:38,329 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:389850x0, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 17:53:38,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:38985-0x1004ed10fed0000 connected 2023-02-13 17:53:38,360 DEBUG [Listener at localhost/43465] zookeeper.ZKUtil(164): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 17:53:38,361 DEBUG [Listener at localhost/43465] zookeeper.ZKUtil(164): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 17:53:38,364 DEBUG [Listener at localhost/43465] zookeeper.ZKUtil(164): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 17:53:38,375 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38985 2023-02-13 17:53:38,375 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38985 2023-02-13 17:53:38,375 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38985 2023-02-13 17:53:38,376 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38985 2023-02-13 17:53:38,376 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38985 2023-02-13 17:53:38,382 INFO [Listener at localhost/43465] master.HMaster(439): hbase.rootdir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2, hbase.cluster.distributed=false 2023-02-13 17:53:38,450 INFO [Listener at localhost/43465] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 17:53:38,451 INFO [Listener at localhost/43465] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:38,451 INFO [Listener at localhost/43465] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:38,451 INFO [Listener at localhost/43465] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 17:53:38,451 INFO [Listener at localhost/43465] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:38,451 INFO [Listener at localhost/43465] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 17:53:38,456 INFO [Listener at localhost/43465] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 17:53:38,459 INFO [Listener at localhost/43465] ipc.NettyRpcServer(120): Bind to /172.31.14.131:35949 2023-02-13 17:53:38,462 INFO [Listener at localhost/43465] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 17:53:38,468 DEBUG [Listener at localhost/43465] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 17:53:38,469 INFO [Listener at localhost/43465] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:38,471 INFO [Listener at localhost/43465] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:38,473 INFO [Listener at localhost/43465] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:35949 connecting to ZooKeeper ensemble=127.0.0.1:52890 2023-02-13 17:53:38,479 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:359490x0, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 17:53:38,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:35949-0x1004ed10fed0001 connected 2023-02-13 17:53:38,480 DEBUG [Listener at localhost/43465] zookeeper.ZKUtil(164): regionserver:35949-0x1004ed10fed0001, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 17:53:38,481 DEBUG [Listener at localhost/43465] zookeeper.ZKUtil(164): regionserver:35949-0x1004ed10fed0001, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 17:53:38,482 DEBUG [Listener at localhost/43465] zookeeper.ZKUtil(164): regionserver:35949-0x1004ed10fed0001, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 17:53:38,483 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35949 2023-02-13 17:53:38,484 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35949 2023-02-13 17:53:38,484 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35949 2023-02-13 17:53:38,486 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35949 2023-02-13 17:53:38,486 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35949 2023-02-13 17:53:38,498 INFO [Listener at localhost/43465] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 17:53:38,499 INFO [Listener at localhost/43465] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:38,499 INFO [Listener at localhost/43465] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:38,500 INFO [Listener at localhost/43465] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 17:53:38,500 INFO [Listener at localhost/43465] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:38,500 INFO [Listener at localhost/43465] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 17:53:38,500 INFO [Listener at localhost/43465] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 17:53:38,504 INFO [Listener at localhost/43465] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39175 2023-02-13 17:53:38,504 INFO [Listener at localhost/43465] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 17:53:38,506 DEBUG [Listener at localhost/43465] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 17:53:38,507 INFO [Listener at localhost/43465] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:38,509 INFO [Listener at localhost/43465] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:38,512 INFO [Listener at localhost/43465] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39175 connecting to ZooKeeper ensemble=127.0.0.1:52890 2023-02-13 17:53:38,517 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:391750x0, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 17:53:38,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:39175-0x1004ed10fed0002 connected 2023-02-13 17:53:38,518 DEBUG [Listener at localhost/43465] zookeeper.ZKUtil(164): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 17:53:38,519 DEBUG [Listener at localhost/43465] zookeeper.ZKUtil(164): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 17:53:38,520 DEBUG [Listener at localhost/43465] zookeeper.ZKUtil(164): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 17:53:38,521 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39175 2023-02-13 17:53:38,521 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39175 2023-02-13 17:53:38,524 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39175 2023-02-13 17:53:38,527 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39175 2023-02-13 17:53:38,527 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39175 2023-02-13 17:53:38,539 INFO [Listener at localhost/43465] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 17:53:38,540 INFO [Listener at localhost/43465] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:38,540 INFO [Listener at localhost/43465] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:38,540 INFO [Listener at localhost/43465] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 17:53:38,540 INFO [Listener at localhost/43465] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:38,540 INFO [Listener at localhost/43465] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 17:53:38,540 INFO [Listener at localhost/43465] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 17:53:38,542 INFO [Listener at localhost/43465] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34325 2023-02-13 17:53:38,542 INFO [Listener at localhost/43465] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 17:53:38,545 DEBUG [Listener at localhost/43465] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 17:53:38,546 INFO [Listener at localhost/43465] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:38,548 INFO [Listener at localhost/43465] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:38,549 INFO [Listener at localhost/43465] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34325 connecting to ZooKeeper ensemble=127.0.0.1:52890 2023-02-13 17:53:38,555 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:343250x0, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 17:53:38,556 DEBUG [Listener at localhost/43465] zookeeper.ZKUtil(164): regionserver:343250x0, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 17:53:38,557 DEBUG [Listener at localhost/43465] zookeeper.ZKUtil(164): regionserver:343250x0, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 17:53:38,558 DEBUG [Listener at localhost/43465] zookeeper.ZKUtil(164): regionserver:343250x0, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 17:53:38,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:34325-0x1004ed10fed0003 connected 2023-02-13 17:53:38,563 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34325 2023-02-13 17:53:38,563 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34325 2023-02-13 17:53:38,568 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34325 2023-02-13 17:53:38,568 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34325 2023-02-13 17:53:38,572 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34325 2023-02-13 17:53:38,574 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,38985,1676310817175 2023-02-13 17:53:38,583 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 17:53:38,585 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,38985,1676310817175 2023-02-13 17:53:38,602 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 17:53:38,602 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 17:53:38,602 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:53:38,602 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:35949-0x1004ed10fed0001, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 17:53:38,602 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:34325-0x1004ed10fed0003, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 17:53:38,604 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 17:53:38,605 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,38985,1676310817175 from backup master directory 2023-02-13 17:53:38,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 17:53:38,609 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,38985,1676310817175 2023-02-13 17:53:38,609 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 17:53:38,610 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 17:53:38,610 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,38985,1676310817175 2023-02-13 17:53:38,613 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-02-13 17:53:38,615 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-02-13 17:53:38,725 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/hbase.id with ID: 16ef12d6-73c5-4907-bcd9-c35fc8da946d 2023-02-13 17:53:38,771 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:38,788 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:53:38,838 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x3150d8d9 to 127.0.0.1:52890 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:53:38,870 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69ee0746, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:53:38,894 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 17:53:38,896 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-13 17:53:38,913 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-02-13 17:53:38,913 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-02-13 17:53:38,915 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-13 17:53:38,919 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-13 17:53:38,920 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 17:53:38,957 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/data/master/store-tmp 2023-02-13 17:53:39,016 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:39,017 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 17:53:39,017 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:53:39,017 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:53:39,017 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 17:53:39,017 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:53:39,017 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:53:39,018 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676310819016Disabling compacts and flushes for region at 1676310819017 (+1 ms)Disabling writes for close at 1676310819017Writing region close event to WAL at 1676310819017Closed at 1676310819017 2023-02-13 17:53:39,019 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/WALs/jenkins-hbase4.apache.org,38985,1676310817175 2023-02-13 17:53:39,043 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38985%2C1676310817175, suffix=, logDir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/WALs/jenkins-hbase4.apache.org,38985,1676310817175, archiveDir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/oldWALs, maxLogs=10 2023-02-13 17:53:39,120 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33739,DS-07b12a43-1f57-44a8-a528-3a10ed60ab98,DISK] 2023-02-13 17:53:39,120 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33275,DS-6ebf9491-78fa-49e0-8c5b-82ee1005d4af,DISK] 2023-02-13 17:53:39,120 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45833,DS-0de4ecf6-a9b9-4f55-81ee-f85a32d928d3,DISK] 2023-02-13 17:53:39,128 DEBUG [RS-EventLoopGroup-5-1] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 17:53:39,211 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/WALs/jenkins-hbase4.apache.org,38985,1676310817175/jenkins-hbase4.apache.org%2C38985%2C1676310817175.1676310819057 2023-02-13 17:53:39,211 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33275,DS-6ebf9491-78fa-49e0-8c5b-82ee1005d4af,DISK], DatanodeInfoWithStorage[127.0.0.1:33739,DS-07b12a43-1f57-44a8-a528-3a10ed60ab98,DISK], DatanodeInfoWithStorage[127.0.0.1:45833,DS-0de4ecf6-a9b9-4f55-81ee-f85a32d928d3,DISK]] 2023-02-13 17:53:39,212 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-13 17:53:39,212 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:39,216 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 17:53:39,217 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 17:53:39,275 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-13 17:53:39,282 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-13 17:53:39,308 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-13 17:53:39,320 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:39,327 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-13 17:53:39,328 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-13 17:53:39,343 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 17:53:39,347 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:53:39,348 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63786792, jitterRate=-0.04950273036956787}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:53:39,348 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676310819243Initializing all the Stores at 1676310819246 (+3 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676310819246Cleaning up temporary data from old regions at 1676310819334 (+88 ms)Cleaning up detritus from prior splits at 1676310819335 (+1 ms)Region opened successfully at 1676310819348 (+13 ms) 2023-02-13 17:53:39,350 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-13 17:53:39,372 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-13 17:53:39,372 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-13 17:53:39,375 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-13 17:53:39,378 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-02-13 17:53:39,410 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 32 msec 2023-02-13 17:53:39,410 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-13 17:53:39,434 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-13 17:53:39,439 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-13 17:53:39,468 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-13 17:53:39,473 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 17:53:39,475 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-13 17:53:39,480 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-13 17:53:39,483 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-13 17:53:39,486 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:53:39,487 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-13 17:53:39,488 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-13 17:53:39,500 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-13 17:53:39,504 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 17:53:39,504 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 17:53:39,504 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:35949-0x1004ed10fed0001, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 17:53:39,504 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:34325-0x1004ed10fed0003, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 17:53:39,504 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:53:39,505 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,38985,1676310817175, sessionid=0x1004ed10fed0000, setting cluster-up flag (Was=false) 2023-02-13 17:53:39,529 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:53:39,536 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-13 17:53:39,537 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,38985,1676310817175 2023-02-13 17:53:39,542 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:53:39,548 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-13 17:53:39,550 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,38985,1676310817175 2023-02-13 17:53:39,552 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/.hbase-snapshot/.tmp 2023-02-13 17:53:39,576 INFO [RS:1;jenkins-hbase4:39175] regionserver.HRegionServer(952): ClusterId : 16ef12d6-73c5-4907-bcd9-c35fc8da946d 2023-02-13 17:53:39,576 INFO [RS:2;jenkins-hbase4:34325] regionserver.HRegionServer(952): ClusterId : 16ef12d6-73c5-4907-bcd9-c35fc8da946d 2023-02-13 17:53:39,576 INFO [RS:0;jenkins-hbase4:35949] regionserver.HRegionServer(952): ClusterId : 16ef12d6-73c5-4907-bcd9-c35fc8da946d 2023-02-13 17:53:39,581 DEBUG [RS:1;jenkins-hbase4:39175] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 17:53:39,581 DEBUG [RS:2;jenkins-hbase4:34325] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 17:53:39,581 DEBUG [RS:0;jenkins-hbase4:35949] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 17:53:39,587 DEBUG [RS:2;jenkins-hbase4:34325] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 17:53:39,587 DEBUG [RS:0;jenkins-hbase4:35949] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 17:53:39,587 DEBUG [RS:1;jenkins-hbase4:39175] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 17:53:39,587 DEBUG [RS:0;jenkins-hbase4:35949] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 17:53:39,587 DEBUG [RS:2;jenkins-hbase4:34325] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 17:53:39,587 DEBUG [RS:1;jenkins-hbase4:39175] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 17:53:39,591 DEBUG [RS:0;jenkins-hbase4:35949] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 17:53:39,591 DEBUG [RS:1;jenkins-hbase4:39175] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 17:53:39,591 DEBUG [RS:2;jenkins-hbase4:34325] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 17:53:39,594 DEBUG [RS:0;jenkins-hbase4:35949] zookeeper.ReadOnlyZKClient(139): Connect 0x6c8db8b8 to 127.0.0.1:52890 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:53:39,594 DEBUG [RS:1;jenkins-hbase4:39175] zookeeper.ReadOnlyZKClient(139): Connect 0x72cc9fb0 to 127.0.0.1:52890 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:53:39,594 DEBUG [RS:2;jenkins-hbase4:34325] zookeeper.ReadOnlyZKClient(139): Connect 0x50ab9e36 to 127.0.0.1:52890 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:53:39,621 DEBUG [RS:0;jenkins-hbase4:35949] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@624a5b59, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:53:39,622 DEBUG [RS:2;jenkins-hbase4:34325] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e6c6a53, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:53:39,622 DEBUG [RS:0;jenkins-hbase4:35949] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6675d3c8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 17:53:39,623 DEBUG [RS:2;jenkins-hbase4:34325] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a53f546, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 17:53:39,623 DEBUG [RS:1;jenkins-hbase4:39175] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7c0b6f99, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:53:39,623 DEBUG [RS:1;jenkins-hbase4:39175] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7834f4c0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 17:53:39,654 DEBUG [RS:2;jenkins-hbase4:34325] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:34325 2023-02-13 17:53:39,654 DEBUG [RS:1;jenkins-hbase4:39175] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:39175 2023-02-13 17:53:39,661 DEBUG [RS:0;jenkins-hbase4:35949] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:35949 2023-02-13 17:53:39,664 INFO [RS:2;jenkins-hbase4:34325] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 17:53:39,664 INFO [RS:2;jenkins-hbase4:34325] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 17:53:39,664 DEBUG [RS:2;jenkins-hbase4:34325] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 17:53:39,664 INFO [RS:1;jenkins-hbase4:39175] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 17:53:39,665 INFO [RS:1;jenkins-hbase4:39175] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 17:53:39,664 INFO [RS:0;jenkins-hbase4:35949] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 17:53:39,665 INFO [RS:0;jenkins-hbase4:35949] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 17:53:39,665 DEBUG [RS:0;jenkins-hbase4:35949] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 17:53:39,665 DEBUG [RS:1;jenkins-hbase4:39175] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 17:53:39,669 INFO [RS:0;jenkins-hbase4:35949] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,38985,1676310817175 with isa=jenkins-hbase4.apache.org/172.31.14.131:35949, startcode=1676310818450 2023-02-13 17:53:39,669 INFO [RS:1;jenkins-hbase4:39175] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,38985,1676310817175 with isa=jenkins-hbase4.apache.org/172.31.14.131:39175, startcode=1676310818498 2023-02-13 17:53:39,669 INFO [RS:2;jenkins-hbase4:34325] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,38985,1676310817175 with isa=jenkins-hbase4.apache.org/172.31.14.131:34325, startcode=1676310818539 2023-02-13 17:53:39,693 DEBUG [RS:1;jenkins-hbase4:39175] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 17:53:39,693 DEBUG [RS:0;jenkins-hbase4:35949] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 17:53:39,693 DEBUG [RS:2;jenkins-hbase4:34325] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 17:53:39,710 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-13 17:53:39,722 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 17:53:39,723 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 17:53:39,723 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 17:53:39,723 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 17:53:39,723 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-13 17:53:39,723 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,723 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 17:53:39,723 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,726 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676310849726 2023-02-13 17:53:39,728 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-13 17:53:39,737 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-13 17:53:39,738 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-13 17:53:39,739 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-13 17:53:39,745 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-13 17:53:39,746 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-13 17:53:39,746 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-13 17:53:39,747 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-13 17:53:39,747 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-13 17:53:39,747 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:39,749 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-13 17:53:39,751 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-13 17:53:39,751 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-13 17:53:39,755 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55391, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 17:53:39,756 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-13 17:53:39,755 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46361, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 17:53:39,755 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40241, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 17:53:39,756 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-13 17:53:39,759 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676310819758,5,FailOnTimeoutGroup] 2023-02-13 17:53:39,760 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676310819760,5,FailOnTimeoutGroup] 2023-02-13 17:53:39,760 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:39,760 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-13 17:53:39,762 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:39,762 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:39,768 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38985] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,35949,1676310818450 2023-02-13 17:53:39,769 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38985] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,39175,1676310818498 2023-02-13 17:53:39,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38985] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,34325,1676310818539 2023-02-13 17:53:39,791 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-13 17:53:39,792 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-13 17:53:39,792 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2 2023-02-13 17:53:39,794 DEBUG [RS:0;jenkins-hbase4:35949] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2 2023-02-13 17:53:39,794 DEBUG [RS:2;jenkins-hbase4:34325] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2 2023-02-13 17:53:39,794 DEBUG [RS:0;jenkins-hbase4:35949] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:42429 2023-02-13 17:53:39,794 DEBUG [RS:1;jenkins-hbase4:39175] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2 2023-02-13 17:53:39,795 DEBUG [RS:0;jenkins-hbase4:35949] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 17:53:39,795 DEBUG [RS:2;jenkins-hbase4:34325] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:42429 2023-02-13 17:53:39,795 DEBUG [RS:1;jenkins-hbase4:39175] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:42429 2023-02-13 17:53:39,795 DEBUG [RS:2;jenkins-hbase4:34325] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 17:53:39,795 DEBUG [RS:1;jenkins-hbase4:39175] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 17:53:39,805 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:39,807 DEBUG [RS:1;jenkins-hbase4:39175] zookeeper.ZKUtil(162): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39175,1676310818498 2023-02-13 17:53:39,807 WARN [RS:1;jenkins-hbase4:39175] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 17:53:39,808 INFO [RS:1;jenkins-hbase4:39175] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 17:53:39,807 DEBUG [RS:2;jenkins-hbase4:34325] zookeeper.ZKUtil(162): regionserver:34325-0x1004ed10fed0003, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34325,1676310818539 2023-02-13 17:53:39,809 DEBUG [RS:1;jenkins-hbase4:39175] regionserver.HRegionServer(1947): logDir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/WALs/jenkins-hbase4.apache.org,39175,1676310818498 2023-02-13 17:53:39,809 WARN [RS:2;jenkins-hbase4:34325] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 17:53:39,809 DEBUG [RS:0;jenkins-hbase4:35949] zookeeper.ZKUtil(162): regionserver:35949-0x1004ed10fed0001, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35949,1676310818450 2023-02-13 17:53:39,809 INFO [RS:2;jenkins-hbase4:34325] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 17:53:39,810 WARN [RS:0;jenkins-hbase4:35949] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 17:53:39,811 DEBUG [RS:2;jenkins-hbase4:34325] regionserver.HRegionServer(1947): logDir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/WALs/jenkins-hbase4.apache.org,34325,1676310818539 2023-02-13 17:53:39,811 INFO [RS:0;jenkins-hbase4:35949] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 17:53:39,811 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,34325,1676310818539] 2023-02-13 17:53:39,812 DEBUG [RS:0;jenkins-hbase4:35949] regionserver.HRegionServer(1947): logDir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/WALs/jenkins-hbase4.apache.org,35949,1676310818450 2023-02-13 17:53:39,812 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,39175,1676310818498] 2023-02-13 17:53:39,812 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,35949,1676310818450] 2023-02-13 17:53:39,829 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:39,832 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 17:53:39,832 DEBUG [RS:2;jenkins-hbase4:34325] zookeeper.ZKUtil(162): regionserver:34325-0x1004ed10fed0003, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34325,1676310818539 2023-02-13 17:53:39,832 DEBUG [RS:1;jenkins-hbase4:39175] zookeeper.ZKUtil(162): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34325,1676310818539 2023-02-13 17:53:39,833 DEBUG [RS:0;jenkins-hbase4:35949] zookeeper.ZKUtil(162): regionserver:35949-0x1004ed10fed0001, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34325,1676310818539 2023-02-13 17:53:39,833 DEBUG [RS:2;jenkins-hbase4:34325] zookeeper.ZKUtil(162): regionserver:34325-0x1004ed10fed0003, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39175,1676310818498 2023-02-13 17:53:39,833 DEBUG [RS:1;jenkins-hbase4:39175] zookeeper.ZKUtil(162): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39175,1676310818498 2023-02-13 17:53:39,833 DEBUG [RS:0;jenkins-hbase4:35949] zookeeper.ZKUtil(162): regionserver:35949-0x1004ed10fed0001, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39175,1676310818498 2023-02-13 17:53:39,834 DEBUG [RS:1;jenkins-hbase4:39175] zookeeper.ZKUtil(162): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35949,1676310818450 2023-02-13 17:53:39,834 DEBUG [RS:2;jenkins-hbase4:34325] zookeeper.ZKUtil(162): regionserver:34325-0x1004ed10fed0003, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35949,1676310818450 2023-02-13 17:53:39,834 DEBUG [RS:0;jenkins-hbase4:35949] zookeeper.ZKUtil(162): regionserver:35949-0x1004ed10fed0001, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35949,1676310818450 2023-02-13 17:53:39,835 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/meta/1588230740/info 2023-02-13 17:53:39,835 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 17:53:39,836 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:39,836 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 17:53:39,839 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/meta/1588230740/rep_barrier 2023-02-13 17:53:39,840 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 17:53:39,841 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:39,841 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 17:53:39,843 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/meta/1588230740/table 2023-02-13 17:53:39,844 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 17:53:39,845 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:39,845 DEBUG [RS:0;jenkins-hbase4:35949] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 17:53:39,845 DEBUG [RS:2;jenkins-hbase4:34325] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 17:53:39,845 DEBUG [RS:1;jenkins-hbase4:39175] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 17:53:39,847 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/meta/1588230740 2023-02-13 17:53:39,848 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/meta/1588230740 2023-02-13 17:53:39,852 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 17:53:39,855 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 17:53:39,858 INFO [RS:2;jenkins-hbase4:34325] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 17:53:39,858 INFO [RS:0;jenkins-hbase4:35949] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 17:53:39,858 INFO [RS:1;jenkins-hbase4:39175] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 17:53:39,863 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:53:39,863 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59000682, jitterRate=-0.12082132697105408}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 17:53:39,863 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1676310819830Initializing all the Stores at 1676310819831 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676310819831Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676310819832 (+1 ms)Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676310819832Cleaning up temporary data from old regions at 1676310819850 (+18 ms)Cleaning up detritus from prior splits at 1676310819851 (+1 ms)Region opened successfully at 1676310819863 (+12 ms) 2023-02-13 17:53:39,864 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 17:53:39,864 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 17:53:39,864 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 17:53:39,864 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 17:53:39,864 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 17:53:39,865 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 17:53:39,865 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676310819864Disabling compacts and flushes for region at 1676310819864Disabling writes for close at 1676310819864Writing region close event to WAL at 1676310819865 (+1 ms)Closed at 1676310819865 2023-02-13 17:53:39,872 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-13 17:53:39,872 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-13 17:53:39,883 INFO [RS:2;jenkins-hbase4:34325] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 17:53:39,889 INFO [RS:1;jenkins-hbase4:39175] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 17:53:39,883 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-13 17:53:39,891 INFO [RS:0;jenkins-hbase4:35949] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 17:53:39,895 INFO [RS:0;jenkins-hbase4:35949] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 17:53:39,895 INFO [RS:0;jenkins-hbase4:35949] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:39,895 INFO [RS:1;jenkins-hbase4:39175] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 17:53:39,898 INFO [RS:1;jenkins-hbase4:39175] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:39,900 INFO [RS:0;jenkins-hbase4:35949] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 17:53:39,900 INFO [RS:1;jenkins-hbase4:39175] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 17:53:39,901 INFO [RS:2;jenkins-hbase4:34325] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 17:53:39,902 INFO [RS:2;jenkins-hbase4:34325] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:39,902 INFO [RS:2;jenkins-hbase4:34325] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 17:53:39,912 INFO [RS:2;jenkins-hbase4:34325] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:39,912 INFO [RS:0;jenkins-hbase4:35949] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:39,912 INFO [RS:1;jenkins-hbase4:39175] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:39,912 DEBUG [RS:2;jenkins-hbase4:34325] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,913 DEBUG [RS:0;jenkins-hbase4:35949] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,913 DEBUG [RS:1;jenkins-hbase4:39175] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,914 DEBUG [RS:0;jenkins-hbase4:35949] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,914 DEBUG [RS:1;jenkins-hbase4:39175] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,914 DEBUG [RS:0;jenkins-hbase4:35949] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,914 DEBUG [RS:1;jenkins-hbase4:39175] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,914 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-13 17:53:39,914 DEBUG [RS:1;jenkins-hbase4:39175] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,913 DEBUG [RS:2;jenkins-hbase4:34325] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,914 DEBUG [RS:1;jenkins-hbase4:39175] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,914 DEBUG [RS:0;jenkins-hbase4:35949] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,915 DEBUG [RS:1;jenkins-hbase4:39175] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 17:53:39,915 DEBUG [RS:0;jenkins-hbase4:35949] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,915 DEBUG [RS:1;jenkins-hbase4:39175] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,915 DEBUG [RS:0;jenkins-hbase4:35949] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 17:53:39,915 DEBUG [RS:1;jenkins-hbase4:39175] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,915 DEBUG [RS:0;jenkins-hbase4:35949] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,915 DEBUG [RS:1;jenkins-hbase4:39175] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,915 DEBUG [RS:0;jenkins-hbase4:35949] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,915 DEBUG [RS:1;jenkins-hbase4:39175] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,915 DEBUG [RS:2;jenkins-hbase4:34325] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,915 DEBUG [RS:0;jenkins-hbase4:35949] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,916 DEBUG [RS:2;jenkins-hbase4:34325] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,916 DEBUG [RS:0;jenkins-hbase4:35949] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,916 DEBUG [RS:2;jenkins-hbase4:34325] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,917 DEBUG [RS:2;jenkins-hbase4:34325] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 17:53:39,917 DEBUG [RS:2;jenkins-hbase4:34325] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,917 DEBUG [RS:2;jenkins-hbase4:34325] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,917 DEBUG [RS:2;jenkins-hbase4:34325] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,917 DEBUG [RS:2;jenkins-hbase4:34325] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:39,917 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-13 17:53:39,918 INFO [RS:1;jenkins-hbase4:39175] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:39,918 INFO [RS:1;jenkins-hbase4:39175] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:39,918 INFO [RS:1;jenkins-hbase4:39175] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:39,920 INFO [RS:2;jenkins-hbase4:34325] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:39,920 INFO [RS:2;jenkins-hbase4:34325] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:39,920 INFO [RS:2;jenkins-hbase4:34325] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:39,922 INFO [RS:0;jenkins-hbase4:35949] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:39,922 INFO [RS:0;jenkins-hbase4:35949] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:39,922 INFO [RS:0;jenkins-hbase4:35949] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:39,939 INFO [RS:2;jenkins-hbase4:34325] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 17:53:39,939 INFO [RS:1;jenkins-hbase4:39175] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 17:53:39,939 INFO [RS:0;jenkins-hbase4:35949] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 17:53:39,942 INFO [RS:2;jenkins-hbase4:34325] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34325,1676310818539-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:39,942 INFO [RS:0;jenkins-hbase4:35949] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35949,1676310818450-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:39,942 INFO [RS:1;jenkins-hbase4:39175] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39175,1676310818498-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:39,962 INFO [RS:0;jenkins-hbase4:35949] regionserver.Replication(203): jenkins-hbase4.apache.org,35949,1676310818450 started 2023-02-13 17:53:39,962 INFO [RS:0;jenkins-hbase4:35949] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,35949,1676310818450, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:35949, sessionid=0x1004ed10fed0001 2023-02-13 17:53:39,962 DEBUG [RS:0;jenkins-hbase4:35949] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 17:53:39,962 DEBUG [RS:0;jenkins-hbase4:35949] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,35949,1676310818450 2023-02-13 17:53:39,962 DEBUG [RS:0;jenkins-hbase4:35949] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35949,1676310818450' 2023-02-13 17:53:39,962 DEBUG [RS:0;jenkins-hbase4:35949] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 17:53:39,963 DEBUG [RS:0;jenkins-hbase4:35949] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 17:53:39,963 INFO [RS:2;jenkins-hbase4:34325] regionserver.Replication(203): jenkins-hbase4.apache.org,34325,1676310818539 started 2023-02-13 17:53:39,963 INFO [RS:2;jenkins-hbase4:34325] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,34325,1676310818539, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:34325, sessionid=0x1004ed10fed0003 2023-02-13 17:53:39,963 DEBUG [RS:2;jenkins-hbase4:34325] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 17:53:39,963 DEBUG [RS:2;jenkins-hbase4:34325] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,34325,1676310818539 2023-02-13 17:53:39,964 DEBUG [RS:2;jenkins-hbase4:34325] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34325,1676310818539' 2023-02-13 17:53:39,964 DEBUG [RS:2;jenkins-hbase4:34325] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 17:53:39,964 DEBUG [RS:0;jenkins-hbase4:35949] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 17:53:39,964 DEBUG [RS:0;jenkins-hbase4:35949] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 17:53:39,965 DEBUG [RS:0;jenkins-hbase4:35949] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,35949,1676310818450 2023-02-13 17:53:39,965 DEBUG [RS:0;jenkins-hbase4:35949] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35949,1676310818450' 2023-02-13 17:53:39,965 DEBUG [RS:0;jenkins-hbase4:35949] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 17:53:39,965 DEBUG [RS:2;jenkins-hbase4:34325] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 17:53:39,965 DEBUG [RS:0;jenkins-hbase4:35949] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 17:53:39,965 DEBUG [RS:2;jenkins-hbase4:34325] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 17:53:39,965 DEBUG [RS:2;jenkins-hbase4:34325] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 17:53:39,965 DEBUG [RS:2;jenkins-hbase4:34325] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,34325,1676310818539 2023-02-13 17:53:39,966 DEBUG [RS:2;jenkins-hbase4:34325] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34325,1676310818539' 2023-02-13 17:53:39,966 DEBUG [RS:2;jenkins-hbase4:34325] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 17:53:39,966 DEBUG [RS:0;jenkins-hbase4:35949] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 17:53:39,966 INFO [RS:1;jenkins-hbase4:39175] regionserver.Replication(203): jenkins-hbase4.apache.org,39175,1676310818498 started 2023-02-13 17:53:39,966 INFO [RS:0;jenkins-hbase4:35949] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 17:53:39,966 INFO [RS:1;jenkins-hbase4:39175] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,39175,1676310818498, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:39175, sessionid=0x1004ed10fed0002 2023-02-13 17:53:39,966 DEBUG [RS:2;jenkins-hbase4:34325] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 17:53:39,966 DEBUG [RS:1;jenkins-hbase4:39175] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 17:53:39,966 DEBUG [RS:1;jenkins-hbase4:39175] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,39175,1676310818498 2023-02-13 17:53:39,966 DEBUG [RS:1;jenkins-hbase4:39175] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39175,1676310818498' 2023-02-13 17:53:39,967 DEBUG [RS:1;jenkins-hbase4:39175] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 17:53:39,966 INFO [RS:0;jenkins-hbase4:35949] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 17:53:39,967 DEBUG [RS:2;jenkins-hbase4:34325] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 17:53:39,967 DEBUG [RS:1;jenkins-hbase4:39175] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 17:53:39,967 INFO [RS:2;jenkins-hbase4:34325] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 17:53:39,968 INFO [RS:2;jenkins-hbase4:34325] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 17:53:39,968 DEBUG [RS:1;jenkins-hbase4:39175] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 17:53:39,968 DEBUG [RS:1;jenkins-hbase4:39175] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 17:53:39,968 DEBUG [RS:1;jenkins-hbase4:39175] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,39175,1676310818498 2023-02-13 17:53:39,968 DEBUG [RS:1;jenkins-hbase4:39175] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39175,1676310818498' 2023-02-13 17:53:39,968 DEBUG [RS:1;jenkins-hbase4:39175] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 17:53:39,969 DEBUG [RS:1;jenkins-hbase4:39175] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 17:53:39,969 DEBUG [RS:1;jenkins-hbase4:39175] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 17:53:39,969 INFO [RS:1;jenkins-hbase4:39175] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 17:53:39,969 INFO [RS:1;jenkins-hbase4:39175] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 17:53:40,072 DEBUG [jenkins-hbase4:38985] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-13 17:53:40,076 DEBUG [jenkins-hbase4:38985] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 17:53:40,081 INFO [RS:2;jenkins-hbase4:34325] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34325%2C1676310818539, suffix=, logDir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/WALs/jenkins-hbase4.apache.org,34325,1676310818539, archiveDir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/oldWALs, maxLogs=32 2023-02-13 17:53:40,081 INFO [RS:1;jenkins-hbase4:39175] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39175%2C1676310818498, suffix=, logDir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/WALs/jenkins-hbase4.apache.org,39175,1676310818498, archiveDir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/oldWALs, maxLogs=32 2023-02-13 17:53:40,083 INFO [RS:0;jenkins-hbase4:35949] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35949%2C1676310818450, suffix=, logDir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/WALs/jenkins-hbase4.apache.org,35949,1676310818450, archiveDir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/oldWALs, maxLogs=32 2023-02-13 17:53:40,085 DEBUG [jenkins-hbase4:38985] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 17:53:40,085 DEBUG [jenkins-hbase4:38985] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 17:53:40,085 DEBUG [jenkins-hbase4:38985] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 17:53:40,085 DEBUG [jenkins-hbase4:38985] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 17:53:40,089 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,39175,1676310818498, state=OPENING 2023-02-13 17:53:40,103 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-13 17:53:40,105 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:53:40,112 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 17:53:40,117 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,39175,1676310818498}] 2023-02-13 17:53:40,141 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45833,DS-0de4ecf6-a9b9-4f55-81ee-f85a32d928d3,DISK] 2023-02-13 17:53:40,141 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33739,DS-07b12a43-1f57-44a8-a528-3a10ed60ab98,DISK] 2023-02-13 17:53:40,142 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33275,DS-6ebf9491-78fa-49e0-8c5b-82ee1005d4af,DISK] 2023-02-13 17:53:40,145 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33739,DS-07b12a43-1f57-44a8-a528-3a10ed60ab98,DISK] 2023-02-13 17:53:40,145 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45833,DS-0de4ecf6-a9b9-4f55-81ee-f85a32d928d3,DISK] 2023-02-13 17:53:40,145 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33275,DS-6ebf9491-78fa-49e0-8c5b-82ee1005d4af,DISK] 2023-02-13 17:53:40,147 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33739,DS-07b12a43-1f57-44a8-a528-3a10ed60ab98,DISK] 2023-02-13 17:53:40,147 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45833,DS-0de4ecf6-a9b9-4f55-81ee-f85a32d928d3,DISK] 2023-02-13 17:53:40,147 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33275,DS-6ebf9491-78fa-49e0-8c5b-82ee1005d4af,DISK] 2023-02-13 17:53:40,163 INFO [RS:2;jenkins-hbase4:34325] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/WALs/jenkins-hbase4.apache.org,34325,1676310818539/jenkins-hbase4.apache.org%2C34325%2C1676310818539.1676310820086 2023-02-13 17:53:40,163 INFO [RS:0;jenkins-hbase4:35949] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/WALs/jenkins-hbase4.apache.org,35949,1676310818450/jenkins-hbase4.apache.org%2C35949%2C1676310818450.1676310820087 2023-02-13 17:53:40,164 INFO [RS:1;jenkins-hbase4:39175] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/WALs/jenkins-hbase4.apache.org,39175,1676310818498/jenkins-hbase4.apache.org%2C39175%2C1676310818498.1676310820087 2023-02-13 17:53:40,164 DEBUG [RS:2;jenkins-hbase4:34325] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45833,DS-0de4ecf6-a9b9-4f55-81ee-f85a32d928d3,DISK], DatanodeInfoWithStorage[127.0.0.1:33275,DS-6ebf9491-78fa-49e0-8c5b-82ee1005d4af,DISK], DatanodeInfoWithStorage[127.0.0.1:33739,DS-07b12a43-1f57-44a8-a528-3a10ed60ab98,DISK]] 2023-02-13 17:53:40,164 DEBUG [RS:1;jenkins-hbase4:39175] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33739,DS-07b12a43-1f57-44a8-a528-3a10ed60ab98,DISK], DatanodeInfoWithStorage[127.0.0.1:45833,DS-0de4ecf6-a9b9-4f55-81ee-f85a32d928d3,DISK], DatanodeInfoWithStorage[127.0.0.1:33275,DS-6ebf9491-78fa-49e0-8c5b-82ee1005d4af,DISK]] 2023-02-13 17:53:40,164 DEBUG [RS:0;jenkins-hbase4:35949] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33275,DS-6ebf9491-78fa-49e0-8c5b-82ee1005d4af,DISK], DatanodeInfoWithStorage[127.0.0.1:33739,DS-07b12a43-1f57-44a8-a528-3a10ed60ab98,DISK], DatanodeInfoWithStorage[127.0.0.1:45833,DS-0de4ecf6-a9b9-4f55-81ee-f85a32d928d3,DISK]] 2023-02-13 17:53:40,343 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,39175,1676310818498 2023-02-13 17:53:40,345 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 17:53:40,349 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60882, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 17:53:40,364 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-13 17:53:40,365 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 17:53:40,368 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39175%2C1676310818498.meta, suffix=.meta, logDir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/WALs/jenkins-hbase4.apache.org,39175,1676310818498, archiveDir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/oldWALs, maxLogs=32 2023-02-13 17:53:40,387 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33739,DS-07b12a43-1f57-44a8-a528-3a10ed60ab98,DISK] 2023-02-13 17:53:40,387 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45833,DS-0de4ecf6-a9b9-4f55-81ee-f85a32d928d3,DISK] 2023-02-13 17:53:40,389 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33275,DS-6ebf9491-78fa-49e0-8c5b-82ee1005d4af,DISK] 2023-02-13 17:53:40,398 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/WALs/jenkins-hbase4.apache.org,39175,1676310818498/jenkins-hbase4.apache.org%2C39175%2C1676310818498.meta.1676310820369.meta 2023-02-13 17:53:40,399 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33739,DS-07b12a43-1f57-44a8-a528-3a10ed60ab98,DISK], DatanodeInfoWithStorage[127.0.0.1:45833,DS-0de4ecf6-a9b9-4f55-81ee-f85a32d928d3,DISK], DatanodeInfoWithStorage[127.0.0.1:33275,DS-6ebf9491-78fa-49e0-8c5b-82ee1005d4af,DISK]] 2023-02-13 17:53:40,400 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-13 17:53:40,403 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-13 17:53:40,421 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-13 17:53:40,426 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-13 17:53:40,432 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-13 17:53:40,432 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:40,432 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-13 17:53:40,432 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-13 17:53:40,435 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 17:53:40,438 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/meta/1588230740/info 2023-02-13 17:53:40,438 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/meta/1588230740/info 2023-02-13 17:53:40,439 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 17:53:40,440 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:40,440 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 17:53:40,441 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/meta/1588230740/rep_barrier 2023-02-13 17:53:40,441 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/meta/1588230740/rep_barrier 2023-02-13 17:53:40,442 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 17:53:40,443 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:40,443 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 17:53:40,444 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/meta/1588230740/table 2023-02-13 17:53:40,445 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/meta/1588230740/table 2023-02-13 17:53:40,445 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 17:53:40,446 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:40,448 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/meta/1588230740 2023-02-13 17:53:40,451 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/meta/1588230740 2023-02-13 17:53:40,455 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 17:53:40,457 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 17:53:40,458 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73272989, jitterRate=0.09185262024402618}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 17:53:40,459 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676310820433Writing region info on filesystem at 1676310820433Initializing all the Stores at 1676310820435 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676310820435Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676310820435Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676310820435Cleaning up temporary data from old regions at 1676310820453 (+18 ms)Cleaning up detritus from prior splits at 1676310820454 (+1 ms)Running coprocessor post-open hooks at 1676310820459 (+5 ms)Region opened successfully at 1676310820459 2023-02-13 17:53:40,469 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676310820332 2023-02-13 17:53:40,486 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-13 17:53:40,487 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-13 17:53:40,487 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,39175,1676310818498, state=OPEN 2023-02-13 17:53:40,490 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-13 17:53:40,490 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 17:53:40,494 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-13 17:53:40,494 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,39175,1676310818498 in 373 msec 2023-02-13 17:53:40,499 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-13 17:53:40,500 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 612 msec 2023-02-13 17:53:40,507 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 884 msec 2023-02-13 17:53:40,507 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676310818598, completionTime=-1 2023-02-13 17:53:40,507 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-13 17:53:40,507 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-13 17:53:40,568 DEBUG [hconnection-0x24b921ee-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:53:40,571 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60898, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:53:40,587 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-13 17:53:40,588 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676310880588 2023-02-13 17:53:40,588 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676310940588 2023-02-13 17:53:40,588 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 80 msec 2023-02-13 17:53:40,618 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38985,1676310817175-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:40,618 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38985,1676310817175-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:40,618 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38985,1676310817175-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:40,620 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:38985, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:40,621 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:40,627 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-13 17:53:40,635 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-13 17:53:40,636 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-13 17:53:40,645 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-13 17:53:40,647 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 17:53:40,650 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 17:53:40,671 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/.tmp/data/hbase/namespace/673bbd347d5066108d814a6b9fc9e57c 2023-02-13 17:53:40,673 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/.tmp/data/hbase/namespace/673bbd347d5066108d814a6b9fc9e57c empty. 2023-02-13 17:53:40,674 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/.tmp/data/hbase/namespace/673bbd347d5066108d814a6b9fc9e57c 2023-02-13 17:53:40,674 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-13 17:53:40,707 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-13 17:53:40,710 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => 673bbd347d5066108d814a6b9fc9e57c, NAME => 'hbase:namespace,,1676310820635.673bbd347d5066108d814a6b9fc9e57c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/.tmp 2023-02-13 17:53:40,729 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676310820635.673bbd347d5066108d814a6b9fc9e57c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:40,729 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing 673bbd347d5066108d814a6b9fc9e57c, disabling compactions & flushes 2023-02-13 17:53:40,729 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676310820635.673bbd347d5066108d814a6b9fc9e57c. 2023-02-13 17:53:40,729 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676310820635.673bbd347d5066108d814a6b9fc9e57c. 2023-02-13 17:53:40,729 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676310820635.673bbd347d5066108d814a6b9fc9e57c. after waiting 0 ms 2023-02-13 17:53:40,729 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676310820635.673bbd347d5066108d814a6b9fc9e57c. 2023-02-13 17:53:40,729 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676310820635.673bbd347d5066108d814a6b9fc9e57c. 2023-02-13 17:53:40,729 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for 673bbd347d5066108d814a6b9fc9e57c: Waiting for close lock at 1676310820729Disabling compacts and flushes for region at 1676310820729Disabling writes for close at 1676310820729Writing region close event to WAL at 1676310820729Closed at 1676310820729 2023-02-13 17:53:40,733 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 17:53:40,748 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676310820635.673bbd347d5066108d814a6b9fc9e57c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676310820736"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676310820736"}]},"ts":"1676310820736"} 2023-02-13 17:53:40,773 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 17:53:40,775 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 17:53:40,780 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676310820775"}]},"ts":"1676310820775"} 2023-02-13 17:53:40,783 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-13 17:53:40,787 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 17:53:40,788 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 17:53:40,788 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 17:53:40,788 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 17:53:40,788 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 17:53:40,791 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=673bbd347d5066108d814a6b9fc9e57c, ASSIGN}] 2023-02-13 17:53:40,793 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=673bbd347d5066108d814a6b9fc9e57c, ASSIGN 2023-02-13 17:53:40,795 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=673bbd347d5066108d814a6b9fc9e57c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39175,1676310818498; forceNewPlan=false, retain=false 2023-02-13 17:53:40,947 INFO [jenkins-hbase4:38985] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 17:53:40,948 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=673bbd347d5066108d814a6b9fc9e57c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39175,1676310818498 2023-02-13 17:53:40,948 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676310820635.673bbd347d5066108d814a6b9fc9e57c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676310820948"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676310820948"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676310820948"}]},"ts":"1676310820948"} 2023-02-13 17:53:40,952 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 673bbd347d5066108d814a6b9fc9e57c, server=jenkins-hbase4.apache.org,39175,1676310818498}] 2023-02-13 17:53:41,113 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676310820635.673bbd347d5066108d814a6b9fc9e57c. 2023-02-13 17:53:41,116 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 673bbd347d5066108d814a6b9fc9e57c, NAME => 'hbase:namespace,,1676310820635.673bbd347d5066108d814a6b9fc9e57c.', STARTKEY => '', ENDKEY => ''} 2023-02-13 17:53:41,117 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 673bbd347d5066108d814a6b9fc9e57c 2023-02-13 17:53:41,117 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676310820635.673bbd347d5066108d814a6b9fc9e57c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:41,117 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 673bbd347d5066108d814a6b9fc9e57c 2023-02-13 17:53:41,117 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 673bbd347d5066108d814a6b9fc9e57c 2023-02-13 17:53:41,119 INFO [StoreOpener-673bbd347d5066108d814a6b9fc9e57c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 673bbd347d5066108d814a6b9fc9e57c 2023-02-13 17:53:41,121 DEBUG [StoreOpener-673bbd347d5066108d814a6b9fc9e57c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/namespace/673bbd347d5066108d814a6b9fc9e57c/info 2023-02-13 17:53:41,121 DEBUG [StoreOpener-673bbd347d5066108d814a6b9fc9e57c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/namespace/673bbd347d5066108d814a6b9fc9e57c/info 2023-02-13 17:53:41,122 INFO [StoreOpener-673bbd347d5066108d814a6b9fc9e57c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 673bbd347d5066108d814a6b9fc9e57c columnFamilyName info 2023-02-13 17:53:41,123 INFO [StoreOpener-673bbd347d5066108d814a6b9fc9e57c-1] regionserver.HStore(310): Store=673bbd347d5066108d814a6b9fc9e57c/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:41,124 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/namespace/673bbd347d5066108d814a6b9fc9e57c 2023-02-13 17:53:41,125 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/namespace/673bbd347d5066108d814a6b9fc9e57c 2023-02-13 17:53:41,130 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 673bbd347d5066108d814a6b9fc9e57c 2023-02-13 17:53:41,133 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/namespace/673bbd347d5066108d814a6b9fc9e57c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:53:41,134 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 673bbd347d5066108d814a6b9fc9e57c; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74482698, jitterRate=0.10987868905067444}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:53:41,134 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 673bbd347d5066108d814a6b9fc9e57c: Running coprocessor pre-open hook at 1676310821117Writing region info on filesystem at 1676310821117Initializing all the Stores at 1676310821118 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676310821118Cleaning up temporary data from old regions at 1676310821126 (+8 ms)Cleaning up detritus from prior splits at 1676310821127 (+1 ms)Running coprocessor post-open hooks at 1676310821134 (+7 ms)Region opened successfully at 1676310821134 2023-02-13 17:53:41,136 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676310820635.673bbd347d5066108d814a6b9fc9e57c., pid=6, masterSystemTime=1676310821105 2023-02-13 17:53:41,139 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676310820635.673bbd347d5066108d814a6b9fc9e57c. 2023-02-13 17:53:41,140 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676310820635.673bbd347d5066108d814a6b9fc9e57c. 2023-02-13 17:53:41,141 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=673bbd347d5066108d814a6b9fc9e57c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39175,1676310818498 2023-02-13 17:53:41,142 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676310820635.673bbd347d5066108d814a6b9fc9e57c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676310821141"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676310821141"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676310821141"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676310821141"}]},"ts":"1676310821141"} 2023-02-13 17:53:41,150 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-13 17:53:41,150 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 673bbd347d5066108d814a6b9fc9e57c, server=jenkins-hbase4.apache.org,39175,1676310818498 in 194 msec 2023-02-13 17:53:41,154 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-13 17:53:41,154 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=673bbd347d5066108d814a6b9fc9e57c, ASSIGN in 359 msec 2023-02-13 17:53:41,155 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 17:53:41,156 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676310821156"}]},"ts":"1676310821156"} 2023-02-13 17:53:41,158 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-13 17:53:41,161 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 17:53:41,164 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 524 msec 2023-02-13 17:53:41,248 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-13 17:53:41,250 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-13 17:53:41,250 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:53:41,287 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-13 17:53:41,305 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-13 17:53:41,314 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 34 msec 2023-02-13 17:53:41,322 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-13 17:53:41,339 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-13 17:53:41,346 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 24 msec 2023-02-13 17:53:41,360 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-13 17:53:41,364 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-13 17:53:41,364 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.754sec 2023-02-13 17:53:41,367 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-13 17:53:41,369 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-13 17:53:41,369 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-13 17:53:41,370 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38985,1676310817175-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-13 17:53:41,372 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38985,1676310817175-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-13 17:53:41,382 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-13 17:53:41,382 DEBUG [Listener at localhost/43465] zookeeper.ReadOnlyZKClient(139): Connect 0x26fb0c0f to 127.0.0.1:52890 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:53:41,387 DEBUG [Listener at localhost/43465] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f01f64a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:53:41,402 DEBUG [hconnection-0x7dff9cd5-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:53:41,414 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60902, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:53:41,424 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,38985,1676310817175 2023-02-13 17:53:41,430 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/balancer 2023-02-13 17:53:41,430 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:53:41,431 INFO [Listener at localhost/43465] master.MasterRpcServices(492): Client=null/null set balanceSwitch=false 2023-02-13 17:53:41,431 INFO [Listener at localhost/43465] master.TestRetainAssignmentOnRestart(212): Creating tables 2023-02-13 17:53:41,441 DEBUG [Listener at localhost/43465] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-13 17:53:41,444 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59522, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-13 17:53:41,456 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38985] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'restartTableOne', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 17:53:41,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38985] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableOne 2023-02-13 17:53:41,460 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 17:53:41,462 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 17:53:41,464 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38985] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "restartTableOne" procId is: 9 2023-02-13 17:53:41,464 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/.tmp/data/default/restartTableOne/f3a07b2ad33c42a90eb09beaac5e4045 2023-02-13 17:53:41,465 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/.tmp/data/default/restartTableOne/f3a07b2ad33c42a90eb09beaac5e4045 empty. 2023-02-13 17:53:41,467 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/.tmp/data/default/restartTableOne/f3a07b2ad33c42a90eb09beaac5e4045 2023-02-13 17:53:41,467 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived restartTableOne regions 2023-02-13 17:53:41,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38985] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 17:53:41,489 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/.tmp/data/default/restartTableOne/.tabledesc/.tableinfo.0000000001 2023-02-13 17:53:41,492 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(7671): creating {ENCODED => f3a07b2ad33c42a90eb09beaac5e4045, NAME => 'restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableOne', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/.tmp 2023-02-13 17:53:41,509 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:41,509 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1603): Closing f3a07b2ad33c42a90eb09beaac5e4045, disabling compactions & flushes 2023-02-13 17:53:41,510 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045. 2023-02-13 17:53:41,510 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045. 2023-02-13 17:53:41,510 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045. after waiting 0 ms 2023-02-13 17:53:41,510 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045. 2023-02-13 17:53:41,510 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1837): Closed restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045. 2023-02-13 17:53:41,510 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1557): Region close journal for f3a07b2ad33c42a90eb09beaac5e4045: Waiting for close lock at 1676310821509Disabling compacts and flushes for region at 1676310821509Disabling writes for close at 1676310821510 (+1 ms)Writing region close event to WAL at 1676310821510Closed at 1676310821510 2023-02-13 17:53:41,516 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 17:53:41,518 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676310821518"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676310821518"}]},"ts":"1676310821518"} 2023-02-13 17:53:41,521 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 17:53:41,523 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 17:53:41,523 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableOne","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676310821523"}]},"ts":"1676310821523"} 2023-02-13 17:53:41,526 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=restartTableOne, state=ENABLING in hbase:meta 2023-02-13 17:53:41,532 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 17:53:41,533 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 17:53:41,533 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 17:53:41,533 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 17:53:41,533 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 17:53:41,534 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=f3a07b2ad33c42a90eb09beaac5e4045, ASSIGN}] 2023-02-13 17:53:41,537 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=f3a07b2ad33c42a90eb09beaac5e4045, ASSIGN 2023-02-13 17:53:41,539 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableOne, region=f3a07b2ad33c42a90eb09beaac5e4045, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35949,1676310818450; forceNewPlan=false, retain=false 2023-02-13 17:53:41,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38985] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 17:53:41,689 INFO [jenkins-hbase4:38985] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 17:53:41,690 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=f3a07b2ad33c42a90eb09beaac5e4045, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35949,1676310818450 2023-02-13 17:53:41,690 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676310821690"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676310821690"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676310821690"}]},"ts":"1676310821690"} 2023-02-13 17:53:41,693 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure f3a07b2ad33c42a90eb09beaac5e4045, server=jenkins-hbase4.apache.org,35949,1676310818450}] 2023-02-13 17:53:41,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38985] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 17:53:41,847 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,35949,1676310818450 2023-02-13 17:53:41,848 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 17:53:41,851 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51030, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 17:53:41,857 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045. 2023-02-13 17:53:41,857 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f3a07b2ad33c42a90eb09beaac5e4045, NAME => 'restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045.', STARTKEY => '', ENDKEY => ''} 2023-02-13 17:53:41,858 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableOne f3a07b2ad33c42a90eb09beaac5e4045 2023-02-13 17:53:41,858 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:41,858 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f3a07b2ad33c42a90eb09beaac5e4045 2023-02-13 17:53:41,858 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f3a07b2ad33c42a90eb09beaac5e4045 2023-02-13 17:53:41,860 INFO [StoreOpener-f3a07b2ad33c42a90eb09beaac5e4045-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region f3a07b2ad33c42a90eb09beaac5e4045 2023-02-13 17:53:41,862 DEBUG [StoreOpener-f3a07b2ad33c42a90eb09beaac5e4045-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableOne/f3a07b2ad33c42a90eb09beaac5e4045/family 2023-02-13 17:53:41,862 DEBUG [StoreOpener-f3a07b2ad33c42a90eb09beaac5e4045-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableOne/f3a07b2ad33c42a90eb09beaac5e4045/family 2023-02-13 17:53:41,863 INFO [StoreOpener-f3a07b2ad33c42a90eb09beaac5e4045-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f3a07b2ad33c42a90eb09beaac5e4045 columnFamilyName family 2023-02-13 17:53:41,863 INFO [StoreOpener-f3a07b2ad33c42a90eb09beaac5e4045-1] regionserver.HStore(310): Store=f3a07b2ad33c42a90eb09beaac5e4045/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:41,865 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableOne/f3a07b2ad33c42a90eb09beaac5e4045 2023-02-13 17:53:41,866 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableOne/f3a07b2ad33c42a90eb09beaac5e4045 2023-02-13 17:53:41,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f3a07b2ad33c42a90eb09beaac5e4045 2023-02-13 17:53:41,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableOne/f3a07b2ad33c42a90eb09beaac5e4045/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:53:41,875 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f3a07b2ad33c42a90eb09beaac5e4045; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65676574, jitterRate=-0.02134278416633606}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:53:41,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f3a07b2ad33c42a90eb09beaac5e4045: Running coprocessor pre-open hook at 1676310821858Writing region info on filesystem at 1676310821858Initializing all the Stores at 1676310821860 (+2 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676310821860Cleaning up temporary data from old regions at 1676310821867 (+7 ms)Cleaning up detritus from prior splits at 1676310821867Running coprocessor post-open hooks at 1676310821875 (+8 ms)Region opened successfully at 1676310821875 2023-02-13 17:53:41,877 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045., pid=11, masterSystemTime=1676310821847 2023-02-13 17:53:41,880 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045. 2023-02-13 17:53:41,881 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045. 2023-02-13 17:53:41,881 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=f3a07b2ad33c42a90eb09beaac5e4045, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35949,1676310818450 2023-02-13 17:53:41,882 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676310821881"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676310821881"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676310821881"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676310821881"}]},"ts":"1676310821881"} 2023-02-13 17:53:41,887 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-02-13 17:53:41,887 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure f3a07b2ad33c42a90eb09beaac5e4045, server=jenkins-hbase4.apache.org,35949,1676310818450 in 191 msec 2023-02-13 17:53:41,890 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-02-13 17:53:41,891 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=restartTableOne, region=f3a07b2ad33c42a90eb09beaac5e4045, ASSIGN in 353 msec 2023-02-13 17:53:41,892 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 17:53:41,892 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableOne","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676310821892"}]},"ts":"1676310821892"} 2023-02-13 17:53:41,894 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=restartTableOne, state=ENABLED in hbase:meta 2023-02-13 17:53:41,897 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 17:53:41,899 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=restartTableOne in 441 msec 2023-02-13 17:53:42,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38985] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 17:53:42,085 INFO [Listener at localhost/43465] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableOne, procId: 9 completed 2023-02-13 17:53:42,085 DEBUG [Listener at localhost/43465] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableOne get assigned. Timeout = 60000ms 2023-02-13 17:53:42,087 INFO [Listener at localhost/43465] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:42,090 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(3484): All regions for table restartTableOne assigned to meta. Checking AM states. 2023-02-13 17:53:42,091 INFO [Listener at localhost/43465] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:42,091 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(3504): All regions for table restartTableOne assigned. 2023-02-13 17:53:42,093 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38985] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'restartTableTwo', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 17:53:42,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38985] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableTwo 2023-02-13 17:53:42,097 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 17:53:42,097 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38985] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "restartTableTwo" procId is: 12 2023-02-13 17:53:42,099 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 17:53:42,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38985] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 17:53:42,101 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/.tmp/data/default/restartTableTwo/dc2f11e16ffd6ee6c6b4b51c20c090b8 2023-02-13 17:53:42,102 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/.tmp/data/default/restartTableTwo/dc2f11e16ffd6ee6c6b4b51c20c090b8 empty. 2023-02-13 17:53:42,103 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/.tmp/data/default/restartTableTwo/dc2f11e16ffd6ee6c6b4b51c20c090b8 2023-02-13 17:53:42,103 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived restartTableTwo regions 2023-02-13 17:53:42,125 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/.tmp/data/default/restartTableTwo/.tabledesc/.tableinfo.0000000001 2023-02-13 17:53:42,126 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(7671): creating {ENCODED => dc2f11e16ffd6ee6c6b4b51c20c090b8, NAME => 'restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableTwo', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/.tmp 2023-02-13 17:53:42,143 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:42,143 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1603): Closing dc2f11e16ffd6ee6c6b4b51c20c090b8, disabling compactions & flushes 2023-02-13 17:53:42,143 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8. 2023-02-13 17:53:42,143 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8. 2023-02-13 17:53:42,143 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8. after waiting 0 ms 2023-02-13 17:53:42,143 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8. 2023-02-13 17:53:42,143 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8. 2023-02-13 17:53:42,143 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1557): Region close journal for dc2f11e16ffd6ee6c6b4b51c20c090b8: Waiting for close lock at 1676310822143Disabling compacts and flushes for region at 1676310822143Disabling writes for close at 1676310822143Writing region close event to WAL at 1676310822143Closed at 1676310822143 2023-02-13 17:53:42,148 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 17:53:42,149 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676310822149"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676310822149"}]},"ts":"1676310822149"} 2023-02-13 17:53:42,152 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 17:53:42,153 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 17:53:42,154 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableTwo","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676310822154"}]},"ts":"1676310822154"} 2023-02-13 17:53:42,156 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=restartTableTwo, state=ENABLING in hbase:meta 2023-02-13 17:53:42,163 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 17:53:42,164 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 17:53:42,164 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 17:53:42,165 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 17:53:42,165 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 17:53:42,165 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=dc2f11e16ffd6ee6c6b4b51c20c090b8, ASSIGN}] 2023-02-13 17:53:42,167 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=dc2f11e16ffd6ee6c6b4b51c20c090b8, ASSIGN 2023-02-13 17:53:42,168 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableTwo, region=dc2f11e16ffd6ee6c6b4b51c20c090b8, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35949,1676310818450; forceNewPlan=false, retain=false 2023-02-13 17:53:42,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38985] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 17:53:42,318 INFO [jenkins-hbase4:38985] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 17:53:42,320 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=dc2f11e16ffd6ee6c6b4b51c20c090b8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35949,1676310818450 2023-02-13 17:53:42,320 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676310822319"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676310822319"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676310822319"}]},"ts":"1676310822319"} 2023-02-13 17:53:42,323 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; OpenRegionProcedure dc2f11e16ffd6ee6c6b4b51c20c090b8, server=jenkins-hbase4.apache.org,35949,1676310818450}] 2023-02-13 17:53:42,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38985] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 17:53:42,480 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8. 2023-02-13 17:53:42,480 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => dc2f11e16ffd6ee6c6b4b51c20c090b8, NAME => 'restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8.', STARTKEY => '', ENDKEY => ''} 2023-02-13 17:53:42,481 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableTwo dc2f11e16ffd6ee6c6b4b51c20c090b8 2023-02-13 17:53:42,481 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:42,481 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for dc2f11e16ffd6ee6c6b4b51c20c090b8 2023-02-13 17:53:42,481 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for dc2f11e16ffd6ee6c6b4b51c20c090b8 2023-02-13 17:53:42,483 INFO [StoreOpener-dc2f11e16ffd6ee6c6b4b51c20c090b8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region dc2f11e16ffd6ee6c6b4b51c20c090b8 2023-02-13 17:53:42,485 DEBUG [StoreOpener-dc2f11e16ffd6ee6c6b4b51c20c090b8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableTwo/dc2f11e16ffd6ee6c6b4b51c20c090b8/family 2023-02-13 17:53:42,485 DEBUG [StoreOpener-dc2f11e16ffd6ee6c6b4b51c20c090b8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableTwo/dc2f11e16ffd6ee6c6b4b51c20c090b8/family 2023-02-13 17:53:42,485 INFO [StoreOpener-dc2f11e16ffd6ee6c6b4b51c20c090b8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region dc2f11e16ffd6ee6c6b4b51c20c090b8 columnFamilyName family 2023-02-13 17:53:42,486 INFO [StoreOpener-dc2f11e16ffd6ee6c6b4b51c20c090b8-1] regionserver.HStore(310): Store=dc2f11e16ffd6ee6c6b4b51c20c090b8/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:42,488 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableTwo/dc2f11e16ffd6ee6c6b4b51c20c090b8 2023-02-13 17:53:42,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableTwo/dc2f11e16ffd6ee6c6b4b51c20c090b8 2023-02-13 17:53:42,494 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for dc2f11e16ffd6ee6c6b4b51c20c090b8 2023-02-13 17:53:42,497 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableTwo/dc2f11e16ffd6ee6c6b4b51c20c090b8/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:53:42,497 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened dc2f11e16ffd6ee6c6b4b51c20c090b8; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60856460, jitterRate=-0.09316807985305786}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:53:42,497 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for dc2f11e16ffd6ee6c6b4b51c20c090b8: Running coprocessor pre-open hook at 1676310822481Writing region info on filesystem at 1676310822481Initializing all the Stores at 1676310822482 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676310822482Cleaning up temporary data from old regions at 1676310822491 (+9 ms)Cleaning up detritus from prior splits at 1676310822491Running coprocessor post-open hooks at 1676310822497 (+6 ms)Region opened successfully at 1676310822497 2023-02-13 17:53:42,499 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8., pid=14, masterSystemTime=1676310822475 2023-02-13 17:53:42,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8. 2023-02-13 17:53:42,501 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8. 2023-02-13 17:53:42,502 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=dc2f11e16ffd6ee6c6b4b51c20c090b8, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35949,1676310818450 2023-02-13 17:53:42,503 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676310822502"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676310822502"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676310822502"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676310822502"}]},"ts":"1676310822502"} 2023-02-13 17:53:42,513 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-02-13 17:53:42,513 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; OpenRegionProcedure dc2f11e16ffd6ee6c6b4b51c20c090b8, server=jenkins-hbase4.apache.org,35949,1676310818450 in 182 msec 2023-02-13 17:53:42,516 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-02-13 17:53:42,517 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=restartTableTwo, region=dc2f11e16ffd6ee6c6b4b51c20c090b8, ASSIGN in 348 msec 2023-02-13 17:53:42,518 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 17:53:42,518 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableTwo","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676310822518"}]},"ts":"1676310822518"} 2023-02-13 17:53:42,520 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=restartTableTwo, state=ENABLED in hbase:meta 2023-02-13 17:53:42,590 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 17:53:42,592 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; CreateTableProcedure table=restartTableTwo in 497 msec 2023-02-13 17:53:42,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38985] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 17:53:42,705 INFO [Listener at localhost/43465] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableTwo, procId: 12 completed 2023-02-13 17:53:42,705 DEBUG [Listener at localhost/43465] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableTwo get assigned. Timeout = 60000ms 2023-02-13 17:53:42,705 INFO [Listener at localhost/43465] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:42,709 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(3484): All regions for table restartTableTwo assigned to meta. Checking AM states. 2023-02-13 17:53:42,709 INFO [Listener at localhost/43465] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:42,709 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(3504): All regions for table restartTableTwo assigned. 2023-02-13 17:53:42,711 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38985] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'restartTableThree', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 17:53:42,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38985] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableThree 2023-02-13 17:53:42,714 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 17:53:42,715 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38985] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "restartTableThree" procId is: 15 2023-02-13 17:53:42,716 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 17:53:42,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38985] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 17:53:42,718 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/.tmp/data/default/restartTableThree/f1791fa38e7b82b917ae39ff48365a4d 2023-02-13 17:53:42,719 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/.tmp/data/default/restartTableThree/f1791fa38e7b82b917ae39ff48365a4d empty. 2023-02-13 17:53:42,719 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/.tmp/data/default/restartTableThree/f1791fa38e7b82b917ae39ff48365a4d 2023-02-13 17:53:42,719 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived restartTableThree regions 2023-02-13 17:53:42,739 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/.tmp/data/default/restartTableThree/.tabledesc/.tableinfo.0000000001 2023-02-13 17:53:42,741 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(7671): creating {ENCODED => f1791fa38e7b82b917ae39ff48365a4d, NAME => 'restartTableThree,,1676310822711.f1791fa38e7b82b917ae39ff48365a4d.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableThree', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/.tmp 2023-02-13 17:53:42,757 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(865): Instantiated restartTableThree,,1676310822711.f1791fa38e7b82b917ae39ff48365a4d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:42,757 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1603): Closing f1791fa38e7b82b917ae39ff48365a4d, disabling compactions & flushes 2023-02-13 17:53:42,757 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1625): Closing region restartTableThree,,1676310822711.f1791fa38e7b82b917ae39ff48365a4d. 2023-02-13 17:53:42,757 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableThree,,1676310822711.f1791fa38e7b82b917ae39ff48365a4d. 2023-02-13 17:53:42,757 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableThree,,1676310822711.f1791fa38e7b82b917ae39ff48365a4d. after waiting 0 ms 2023-02-13 17:53:42,757 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableThree,,1676310822711.f1791fa38e7b82b917ae39ff48365a4d. 2023-02-13 17:53:42,757 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1837): Closed restartTableThree,,1676310822711.f1791fa38e7b82b917ae39ff48365a4d. 2023-02-13 17:53:42,757 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1557): Region close journal for f1791fa38e7b82b917ae39ff48365a4d: Waiting for close lock at 1676310822757Disabling compacts and flushes for region at 1676310822757Disabling writes for close at 1676310822757Writing region close event to WAL at 1676310822757Closed at 1676310822757 2023-02-13 17:53:42,761 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 17:53:42,763 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableThree,,1676310822711.f1791fa38e7b82b917ae39ff48365a4d.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676310822763"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676310822763"}]},"ts":"1676310822763"} 2023-02-13 17:53:42,766 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 17:53:42,767 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 17:53:42,767 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableThree","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676310822767"}]},"ts":"1676310822767"} 2023-02-13 17:53:42,771 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=restartTableThree, state=ENABLING in hbase:meta 2023-02-13 17:53:42,775 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 17:53:42,777 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 17:53:42,777 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 17:53:42,777 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 17:53:42,777 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 17:53:42,777 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=f1791fa38e7b82b917ae39ff48365a4d, ASSIGN}] 2023-02-13 17:53:42,780 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=f1791fa38e7b82b917ae39ff48365a4d, ASSIGN 2023-02-13 17:53:42,781 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableThree, region=f1791fa38e7b82b917ae39ff48365a4d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39175,1676310818498; forceNewPlan=false, retain=false 2023-02-13 17:53:42,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38985] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 17:53:42,931 INFO [jenkins-hbase4:38985] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 17:53:42,932 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=f1791fa38e7b82b917ae39ff48365a4d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39175,1676310818498 2023-02-13 17:53:42,933 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableThree,,1676310822711.f1791fa38e7b82b917ae39ff48365a4d.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676310822932"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676310822932"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676310822932"}]},"ts":"1676310822932"} 2023-02-13 17:53:42,936 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE; OpenRegionProcedure f1791fa38e7b82b917ae39ff48365a4d, server=jenkins-hbase4.apache.org,39175,1676310818498}] 2023-02-13 17:53:43,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38985] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 17:53:43,094 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open restartTableThree,,1676310822711.f1791fa38e7b82b917ae39ff48365a4d. 2023-02-13 17:53:43,095 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f1791fa38e7b82b917ae39ff48365a4d, NAME => 'restartTableThree,,1676310822711.f1791fa38e7b82b917ae39ff48365a4d.', STARTKEY => '', ENDKEY => ''} 2023-02-13 17:53:43,095 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableThree f1791fa38e7b82b917ae39ff48365a4d 2023-02-13 17:53:43,095 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated restartTableThree,,1676310822711.f1791fa38e7b82b917ae39ff48365a4d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:43,095 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f1791fa38e7b82b917ae39ff48365a4d 2023-02-13 17:53:43,095 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f1791fa38e7b82b917ae39ff48365a4d 2023-02-13 17:53:43,097 INFO [StoreOpener-f1791fa38e7b82b917ae39ff48365a4d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region f1791fa38e7b82b917ae39ff48365a4d 2023-02-13 17:53:43,099 DEBUG [StoreOpener-f1791fa38e7b82b917ae39ff48365a4d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableThree/f1791fa38e7b82b917ae39ff48365a4d/family 2023-02-13 17:53:43,099 DEBUG [StoreOpener-f1791fa38e7b82b917ae39ff48365a4d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableThree/f1791fa38e7b82b917ae39ff48365a4d/family 2023-02-13 17:53:43,100 INFO [StoreOpener-f1791fa38e7b82b917ae39ff48365a4d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f1791fa38e7b82b917ae39ff48365a4d columnFamilyName family 2023-02-13 17:53:43,101 INFO [StoreOpener-f1791fa38e7b82b917ae39ff48365a4d-1] regionserver.HStore(310): Store=f1791fa38e7b82b917ae39ff48365a4d/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:43,102 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableThree/f1791fa38e7b82b917ae39ff48365a4d 2023-02-13 17:53:43,103 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableThree/f1791fa38e7b82b917ae39ff48365a4d 2023-02-13 17:53:43,109 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f1791fa38e7b82b917ae39ff48365a4d 2023-02-13 17:53:43,115 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableThree/f1791fa38e7b82b917ae39ff48365a4d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:53:43,116 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f1791fa38e7b82b917ae39ff48365a4d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70044202, jitterRate=0.04373994469642639}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:53:43,116 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f1791fa38e7b82b917ae39ff48365a4d: Running coprocessor pre-open hook at 1676310823095Writing region info on filesystem at 1676310823095Initializing all the Stores at 1676310823097 (+2 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676310823097Cleaning up temporary data from old regions at 1676310823104 (+7 ms)Cleaning up detritus from prior splits at 1676310823105 (+1 ms)Running coprocessor post-open hooks at 1676310823116 (+11 ms)Region opened successfully at 1676310823116 2023-02-13 17:53:43,117 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableThree,,1676310822711.f1791fa38e7b82b917ae39ff48365a4d., pid=17, masterSystemTime=1676310823089 2023-02-13 17:53:43,119 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableThree,,1676310822711.f1791fa38e7b82b917ae39ff48365a4d. 2023-02-13 17:53:43,119 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened restartTableThree,,1676310822711.f1791fa38e7b82b917ae39ff48365a4d. 2023-02-13 17:53:43,120 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=f1791fa38e7b82b917ae39ff48365a4d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39175,1676310818498 2023-02-13 17:53:43,121 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableThree,,1676310822711.f1791fa38e7b82b917ae39ff48365a4d.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676310823120"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676310823120"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676310823120"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676310823120"}]},"ts":"1676310823120"} 2023-02-13 17:53:43,126 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-02-13 17:53:43,126 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; OpenRegionProcedure f1791fa38e7b82b917ae39ff48365a4d, server=jenkins-hbase4.apache.org,39175,1676310818498 in 187 msec 2023-02-13 17:53:43,128 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-02-13 17:53:43,129 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; TransitRegionStateProcedure table=restartTableThree, region=f1791fa38e7b82b917ae39ff48365a4d, ASSIGN in 349 msec 2023-02-13 17:53:43,130 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 17:53:43,130 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableThree","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676310823130"}]},"ts":"1676310823130"} 2023-02-13 17:53:43,132 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=restartTableThree, state=ENABLED in hbase:meta 2023-02-13 17:53:43,134 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 17:53:43,136 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; CreateTableProcedure table=restartTableThree in 423 msec 2023-02-13 17:53:43,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38985] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 17:53:43,321 INFO [Listener at localhost/43465] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableThree, procId: 15 completed 2023-02-13 17:53:43,321 DEBUG [Listener at localhost/43465] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableThree get assigned. Timeout = 60000ms 2023-02-13 17:53:43,321 INFO [Listener at localhost/43465] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:43,325 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(3484): All regions for table restartTableThree assigned to meta. Checking AM states. 2023-02-13 17:53:43,325 INFO [Listener at localhost/43465] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:43,326 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(3504): All regions for table restartTableThree assigned. 2023-02-13 17:53:43,326 INFO [Listener at localhost/43465] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:43,332 INFO [Listener at localhost/43465] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:43,335 INFO [Listener at localhost/43465] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:43,338 INFO [Listener at localhost/43465] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:43,339 INFO [Listener at localhost/43465] master.SnapshotOfRegionAssignmentFromMeta(102): Start to scan the hbase:meta for the current region assignment snappshot 2023-02-13 17:53:43,348 INFO [Listener at localhost/43465] master.SnapshotOfRegionAssignmentFromMeta(164): Finished to scan the hbase:meta for the current region assignmentsnapshot 2023-02-13 17:53:43,348 INFO [Listener at localhost/43465] master.TestRetainAssignmentOnRestart(157): Stopping HMaster and jenkins-hbase4.apache.org,35949,1676310818450 server 2023-02-13 17:53:43,348 INFO [Listener at localhost/43465] hbase.MiniHBaseCluster(631): Stopping Thread[M:0;jenkins-hbase4:38985,5,FailOnTimeoutGroup] 2023-02-13 17:53:43,348 INFO [Listener at localhost/43465] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,38985,1676310817175' ***** 2023-02-13 17:53:43,348 INFO [Listener at localhost/43465] regionserver.HRegionServer(2310): STOPPED: Stopping master 0 2023-02-13 17:53:43,349 DEBUG [M:0;jenkins-hbase4:38985] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ffb0992, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 17:53:43,349 INFO [M:0;jenkins-hbase4:38985] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,38985,1676310817175 2023-02-13 17:53:43,350 DEBUG [M:0;jenkins-hbase4:38985] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3150d8d9 to 127.0.0.1:52890 2023-02-13 17:53:43,350 DEBUG [M:0;jenkins-hbase4:38985] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:43,352 INFO [M:0;jenkins-hbase4:38985] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,38985,1676310817175; all regions closed. 2023-02-13 17:53:43,352 DEBUG [M:0;jenkins-hbase4:38985] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:43,353 DEBUG [M:0;jenkins-hbase4:38985] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-13 17:53:43,353 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-13 17:53:43,353 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676310819758] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676310819758,5,FailOnTimeoutGroup] 2023-02-13 17:53:43,353 DEBUG [M:0;jenkins-hbase4:38985] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-13 17:53:43,354 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676310819760] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676310819760,5,FailOnTimeoutGroup] 2023-02-13 17:53:43,355 INFO [M:0;jenkins-hbase4:38985] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-13 17:53:43,355 INFO [Listener at localhost/43465] hbase.LocalHBaseCluster(360): Waiting on jenkins-hbase4.apache.org,38985,1676310817175 2023-02-13 17:53:43,355 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 17:53:43,355 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:35949-0x1004ed10fed0001, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 17:53:43,355 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 17:53:43,355 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:34325-0x1004ed10fed0003, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 17:53:43,356 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:53:43,356 INFO [M:0;jenkins-hbase4:38985] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-13 17:53:43,357 INFO [M:0;jenkins-hbase4:38985] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-13 17:53:43,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 17:53:43,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34325-0x1004ed10fed0003, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 17:53:43,357 DEBUG [M:0;jenkins-hbase4:38985] master.HMaster(1502): Stopping service threads 2023-02-13 17:53:43,357 INFO [M:0;jenkins-hbase4:38985] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-13 17:53:43,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 17:53:43,358 INFO [M:0;jenkins-hbase4:38985] procedure2.ProcedureExecutor(629): Stopping 2023-02-13 17:53:43,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35949-0x1004ed10fed0001, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 17:53:43,359 ERROR [M:0;jenkins-hbase4:38985] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] 2023-02-13 17:53:43,359 INFO [M:0;jenkins-hbase4:38985] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-13 17:53:43,359 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-13 17:53:43,360 DEBUG [M:0;jenkins-hbase4:38985] zookeeper.ZKUtil(398): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-13 17:53:43,360 WARN [M:0;jenkins-hbase4:38985] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-13 17:53:43,360 INFO [M:0;jenkins-hbase4:38985] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-13 17:53:43,360 INFO [M:0;jenkins-hbase4:38985] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-13 17:53:43,361 DEBUG [M:0;jenkins-hbase4:38985] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 17:53:43,361 INFO [M:0;jenkins-hbase4:38985] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:53:43,361 DEBUG [M:0;jenkins-hbase4:38985] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:53:43,361 DEBUG [M:0;jenkins-hbase4:38985] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 17:53:43,361 DEBUG [M:0;jenkins-hbase4:38985] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:53:43,362 INFO [M:0;jenkins-hbase4:38985] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=64.21 KB heapSize=76.14 KB 2023-02-13 17:53:43,427 INFO [M:0;jenkins-hbase4:38985] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.21 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/ac056f9f747149e8b3a96b94e282ec18 2023-02-13 17:53:43,465 DEBUG [M:0;jenkins-hbase4:38985] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/ac056f9f747149e8b3a96b94e282ec18 as hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/ac056f9f747149e8b3a96b94e282ec18 2023-02-13 17:53:43,474 INFO [M:0;jenkins-hbase4:38985] regionserver.HStore(1080): Added hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/ac056f9f747149e8b3a96b94e282ec18, entries=17, sequenceid=141, filesize=8.5 K 2023-02-13 17:53:43,475 INFO [M:0;jenkins-hbase4:38985] regionserver.HRegion(2947): Finished flush of dataSize ~64.21 KB/65755, heapSize ~76.13 KB/77952, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 114ms, sequenceid=141, compaction requested=false 2023-02-13 17:53:43,479 INFO [M:0;jenkins-hbase4:38985] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:53:43,479 DEBUG [M:0;jenkins-hbase4:38985] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676310823360Disabling compacts and flushes for region at 1676310823360Disabling writes for close at 1676310823361 (+1 ms)Obtaining lock to block concurrent updates at 1676310823362 (+1 ms)Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676310823362Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=65755, getHeapSize=77952, getOffHeapSize=0, getCellsCount=158 at 1676310823370 (+8 ms)Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676310823371 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676310823373 (+2 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676310823399 (+26 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676310823403 (+4 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676310823467 (+64 ms)Finished flush of dataSize ~64.21 KB/65755, heapSize ~76.13 KB/77952, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 114ms, sequenceid=141, compaction requested=false at 1676310823475 (+8 ms)Writing region close event to WAL at 1676310823479 (+4 ms)Closed at 1676310823479 2023-02-13 17:53:43,485 INFO [M:0;jenkins-hbase4:38985] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-13 17:53:43,485 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 17:53:43,486 INFO [M:0;jenkins-hbase4:38985] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:38985 2023-02-13 17:53:43,493 DEBUG [M:0;jenkins-hbase4:38985] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,38985,1676310817175 already deleted, retry=false 2023-02-13 17:53:43,550 WARN [RS-EventLoopGroup-5-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:38985 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:38985 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 17:53:43,551 DEBUG [RS-EventLoopGroup-5-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:38985 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:38985 2023-02-13 17:53:43,552 DEBUG [RS:2;jenkins-hbase4:34325] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 17:53:43,557 WARN [RS-EventLoopGroup-5-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:38985 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:38985 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 17:53:43,557 DEBUG [RS-EventLoopGroup-5-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:38985 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:38985 2023-02-13 17:53:43,558 DEBUG [RS:0;jenkins-hbase4:35949] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 17:53:43,572 WARN [RS-EventLoopGroup-5-3] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:38985 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:38985 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 17:53:43,573 DEBUG [RS-EventLoopGroup-5-3] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:38985 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:38985 2023-02-13 17:53:43,573 DEBUG [RS:1;jenkins-hbase4:39175] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 17:53:43,595 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:43,596 INFO [M:0;jenkins-hbase4:38985] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,38985,1676310817175; zookeeper connection closed. 2023-02-13 17:53:43,596 ERROR [Listener at localhost/43465-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@b6d9a80 rejected from java.util.concurrent.ThreadPoolExecutor@ee9f958[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 25] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:43,596 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:38985-0x1004ed10fed0000, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:43,596 ERROR [Listener at localhost/43465-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@33d133fb rejected from java.util.concurrent.ThreadPoolExecutor@ee9f958[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 25] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:43,597 INFO [Listener at localhost/43465] hbase.MiniHBaseCluster(508): Stopping Thread[RS:0;jenkins-hbase4:35949,5,FailOnTimeoutGroup] 2023-02-13 17:53:43,597 INFO [Listener at localhost/43465] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,35949,1676310818450' ***** 2023-02-13 17:53:43,597 INFO [Listener at localhost/43465] regionserver.HRegionServer(2310): STOPPED: Stopping rs 0 2023-02-13 17:53:43,597 INFO [Listener at localhost/43465] master.TestRetainAssignmentOnRestart(163): Sleeping a bit 2023-02-13 17:53:43,625 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 17:53:43,758 INFO [RS:0;jenkins-hbase4:35949] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 17:53:43,758 INFO [RS:0;jenkins-hbase4:35949] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 17:53:43,758 INFO [RS:0;jenkins-hbase4:35949] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 17:53:43,758 INFO [RS:0;jenkins-hbase4:35949] regionserver.HRegionServer(3304): Received CLOSE for f3a07b2ad33c42a90eb09beaac5e4045 2023-02-13 17:53:43,760 INFO [RS:0;jenkins-hbase4:35949] regionserver.HRegionServer(3304): Received CLOSE for dc2f11e16ffd6ee6c6b4b51c20c090b8 2023-02-13 17:53:43,761 INFO [RS:0;jenkins-hbase4:35949] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,35949,1676310818450 2023-02-13 17:53:43,761 DEBUG [RS:0;jenkins-hbase4:35949] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6c8db8b8 to 127.0.0.1:52890 2023-02-13 17:53:43,761 DEBUG [RS:0;jenkins-hbase4:35949] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:43,761 INFO [RS:0;jenkins-hbase4:35949] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-13 17:53:43,761 DEBUG [RS:0;jenkins-hbase4:35949] regionserver.HRegionServer(1479): Online Regions={f3a07b2ad33c42a90eb09beaac5e4045=restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045., dc2f11e16ffd6ee6c6b4b51c20c090b8=restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8.} 2023-02-13 17:53:43,762 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f3a07b2ad33c42a90eb09beaac5e4045, disabling compactions & flushes 2023-02-13 17:53:43,762 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045. 2023-02-13 17:53:43,763 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045. 2023-02-13 17:53:43,763 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045. after waiting 0 ms 2023-02-13 17:53:43,763 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045. 2023-02-13 17:53:43,764 DEBUG [RS:0;jenkins-hbase4:35949] regionserver.HRegionServer(1505): Waiting on dc2f11e16ffd6ee6c6b4b51c20c090b8, f3a07b2ad33c42a90eb09beaac5e4045 2023-02-13 17:53:43,770 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableOne/f3a07b2ad33c42a90eb09beaac5e4045/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 17:53:43,771 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045. 2023-02-13 17:53:43,771 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f3a07b2ad33c42a90eb09beaac5e4045: Waiting for close lock at 1676310823761Running coprocessor pre-close hooks at 1676310823761Disabling compacts and flushes for region at 1676310823761Disabling writes for close at 1676310823763 (+2 ms)Writing region close event to WAL at 1676310823763Running coprocessor post-close hooks at 1676310823771 (+8 ms)Closed at 1676310823771 2023-02-13 17:53:43,771 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045. 2023-02-13 17:53:43,772 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing dc2f11e16ffd6ee6c6b4b51c20c090b8, disabling compactions & flushes 2023-02-13 17:53:43,772 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8. 2023-02-13 17:53:43,772 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8. 2023-02-13 17:53:43,772 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8. after waiting 0 ms 2023-02-13 17:53:43,772 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8. 2023-02-13 17:53:43,779 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableTwo/dc2f11e16ffd6ee6c6b4b51c20c090b8/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 17:53:43,780 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8. 2023-02-13 17:53:43,781 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for dc2f11e16ffd6ee6c6b4b51c20c090b8: Waiting for close lock at 1676310823772Running coprocessor pre-close hooks at 1676310823772Disabling compacts and flushes for region at 1676310823772Disabling writes for close at 1676310823772Writing region close event to WAL at 1676310823773 (+1 ms)Running coprocessor post-close hooks at 1676310823780 (+7 ms)Closed at 1676310823780 2023-02-13 17:53:43,781 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8. 2023-02-13 17:53:43,827 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 17:53:43,925 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-13 17:53:43,925 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-13 17:53:43,964 INFO [RS:0;jenkins-hbase4:35949] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,35949,1676310818450; all regions closed. 2023-02-13 17:53:43,982 DEBUG [RS:0;jenkins-hbase4:35949] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/oldWALs 2023-02-13 17:53:43,983 INFO [RS:0;jenkins-hbase4:35949] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C35949%2C1676310818450:(num 1676310820087) 2023-02-13 17:53:43,983 DEBUG [RS:0;jenkins-hbase4:35949] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:43,983 INFO [RS:0;jenkins-hbase4:35949] regionserver.LeaseManager(133): Closed leases 2023-02-13 17:53:43,983 INFO [RS:0;jenkins-hbase4:35949] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 17:53:43,983 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 17:53:43,983 INFO [RS:0;jenkins-hbase4:35949] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 17:53:43,984 INFO [RS:0;jenkins-hbase4:35949] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 17:53:43,984 INFO [RS:0;jenkins-hbase4:35949] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 17:53:43,985 INFO [RS:0;jenkins-hbase4:35949] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:35949 2023-02-13 17:53:43,988 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,35949,1676310818450 2023-02-13 17:53:43,988 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:35949-0x1004ed10fed0001, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,35949,1676310818450 2023-02-13 17:53:43,988 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:43,988 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:35949-0x1004ed10fed0001, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:43,988 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:34325-0x1004ed10fed0003, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,35949,1676310818450 2023-02-13 17:53:43,988 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:34325-0x1004ed10fed0003, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:43,988 ERROR [Listener at localhost/43465-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@14d7fcb4 rejected from java.util.concurrent.ThreadPoolExecutor@10e86af6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:43,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34325,1676310818539 2023-02-13 17:53:43,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34325-0x1004ed10fed0003, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34325,1676310818539 2023-02-13 17:53:43,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39175,1676310818498 2023-02-13 17:53:43,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34325-0x1004ed10fed0003, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39175,1676310818498 2023-02-13 17:53:43,991 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,35949,1676310818450 znode expired, triggering replicatorRemoved event 2023-02-13 17:53:43,993 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,35949,1676310818450 znode expired, triggering replicatorRemoved event 2023-02-13 17:53:43,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34325,1676310818539 2023-02-13 17:53:43,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34325-0x1004ed10fed0003, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34325,1676310818539 2023-02-13 17:53:43,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39175,1676310818498 2023-02-13 17:53:43,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34325-0x1004ed10fed0003, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39175,1676310818498 2023-02-13 17:53:44,090 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:35949-0x1004ed10fed0001, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:44,090 INFO [RS:0;jenkins-hbase4:35949] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,35949,1676310818450; zookeeper connection closed. 2023-02-13 17:53:44,090 ERROR [Listener at localhost/43465-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4f6cc185 rejected from java.util.concurrent.ThreadPoolExecutor@10e86af6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:44,091 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:35949-0x1004ed10fed0001, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:44,091 ERROR [Listener at localhost/43465-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3719cf22 rejected from java.util.concurrent.ThreadPoolExecutor@10e86af6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:44,091 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@7dd1b8f3] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@7dd1b8f3 2023-02-13 17:53:44,556 DEBUG [RS:2;jenkins-hbase4:34325] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 17:53:44,577 DEBUG [RS:1;jenkins-hbase4:39175] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 17:53:45,559 DEBUG [RS:2;jenkins-hbase4:34325] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 17:53:45,580 DEBUG [RS:1;jenkins-hbase4:39175] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 17:53:45,598 INFO [Listener at localhost/43465] master.TestRetainAssignmentOnRestart(166): Starting HMaster and region server jenkins-hbase4.apache.org,35949,1676310818450 second time with the same port 2023-02-13 17:53:45,610 INFO [Listener at localhost/43465] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 17:53:45,610 INFO [Listener at localhost/43465] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:45,610 INFO [Listener at localhost/43465] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:45,611 INFO [Listener at localhost/43465] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 17:53:45,611 INFO [Listener at localhost/43465] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:45,611 INFO [Listener at localhost/43465] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 17:53:45,611 INFO [Listener at localhost/43465] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 17:53:45,613 INFO [Listener at localhost/43465] ipc.NettyRpcServer(120): Bind to /172.31.14.131:37439 2023-02-13 17:53:45,614 INFO [Listener at localhost/43465] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:45,615 INFO [Listener at localhost/43465] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:45,616 INFO [Listener at localhost/43465] zookeeper.RecoverableZooKeeper(93): Process identifier=master:37439 connecting to ZooKeeper ensemble=127.0.0.1:52890 2023-02-13 17:53:45,619 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:374390x0, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 17:53:45,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:37439-0x1004ed10fed0009 connected 2023-02-13 17:53:45,624 DEBUG [Listener at localhost/43465] zookeeper.ZKUtil(164): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 17:53:45,625 DEBUG [Listener at localhost/43465] zookeeper.ZKUtil(162): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-13 17:53:45,626 DEBUG [Listener at localhost/43465] zookeeper.ZKUtil(164): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 17:53:45,629 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37439 2023-02-13 17:53:45,629 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37439 2023-02-13 17:53:45,629 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37439 2023-02-13 17:53:45,630 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37439 2023-02-13 17:53:45,630 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37439 2023-02-13 17:53:45,630 INFO [Listener at localhost/43465] master.HMaster(439): hbase.rootdir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2, hbase.cluster.distributed=false 2023-02-13 17:53:45,635 DEBUG [pool-218-thread-1] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: INIT 2023-02-13 17:53:45,636 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,37439,1676310825610 2023-02-13 17:53:45,642 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 17:53:45,642 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,37439,1676310825610 2023-02-13 17:53:45,644 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 17:53:45,644 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:34325-0x1004ed10fed0003, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 17:53:45,644 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:53:45,644 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 17:53:45,648 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 17:53:45,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 17:53:45,649 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,37439,1676310825610 from backup master directory 2023-02-13 17:53:45,649 INFO [M:0;jenkins-hbase4:37439] regionserver.HRegionServer(952): ClusterId : 16ef12d6-73c5-4907-bcd9-c35fc8da946d 2023-02-13 17:53:45,650 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,37439,1676310825610 2023-02-13 17:53:45,650 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 17:53:45,650 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 17:53:45,650 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,37439,1676310825610 2023-02-13 17:53:45,653 INFO [Listener at localhost/43465] client.ConnectionUtils(127): regionserver/jenkins-hbase4:35949 server-side Connection retries=45 2023-02-13 17:53:45,653 INFO [Listener at localhost/43465] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:45,653 INFO [Listener at localhost/43465] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:45,653 INFO [Listener at localhost/43465] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 17:53:45,653 INFO [Listener at localhost/43465] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:45,654 INFO [Listener at localhost/43465] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 17:53:45,654 INFO [Listener at localhost/43465] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 17:53:45,660 INFO [Listener at localhost/43465] ipc.NettyRpcServer(120): Bind to /172.31.14.131:35949 2023-02-13 17:53:45,661 INFO [Listener at localhost/43465] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 17:53:45,662 DEBUG [Listener at localhost/43465] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 17:53:45,663 INFO [Listener at localhost/43465] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:45,665 INFO [Listener at localhost/43465] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:45,666 INFO [Listener at localhost/43465] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:35949 connecting to ZooKeeper ensemble=127.0.0.1:52890 2023-02-13 17:53:45,671 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:359490x0, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 17:53:45,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:35949-0x1004ed10fed000a connected 2023-02-13 17:53:45,673 DEBUG [Listener at localhost/43465] zookeeper.ZKUtil(162): regionserver:35949-0x1004ed10fed000a, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 17:53:45,674 DEBUG [Listener at localhost/43465] zookeeper.ZKUtil(162): regionserver:35949-0x1004ed10fed000a, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-13 17:53:45,675 DEBUG [Listener at localhost/43465] zookeeper.ZKUtil(164): regionserver:35949-0x1004ed10fed000a, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 17:53:45,676 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35949 2023-02-13 17:53:45,680 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35949 2023-02-13 17:53:45,681 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35949 2023-02-13 17:53:45,682 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35949 2023-02-13 17:53:45,682 DEBUG [Listener at localhost/43465] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35949 2023-02-13 17:53:45,684 INFO [RS:3;jenkins-hbase4:35949] regionserver.HRegionServer(952): ClusterId : 16ef12d6-73c5-4907-bcd9-c35fc8da946d 2023-02-13 17:53:45,685 DEBUG [RS:3;jenkins-hbase4:35949] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 17:53:45,689 DEBUG [RS:3;jenkins-hbase4:35949] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 17:53:45,689 DEBUG [RS:3;jenkins-hbase4:35949] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 17:53:45,693 DEBUG [RS:3;jenkins-hbase4:35949] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 17:53:45,694 DEBUG [RS:3;jenkins-hbase4:35949] zookeeper.ReadOnlyZKClient(139): Connect 0x64cbde09 to 127.0.0.1:52890 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:53:45,698 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:45,713 DEBUG [RS:3;jenkins-hbase4:35949] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a0c3ce0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:53:45,713 DEBUG [RS:3;jenkins-hbase4:35949] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6e33b579, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 17:53:45,727 DEBUG [RS:3;jenkins-hbase4:35949] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:3;jenkins-hbase4:35949 2023-02-13 17:53:45,727 INFO [RS:3;jenkins-hbase4:35949] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 17:53:45,727 INFO [RS:3;jenkins-hbase4:35949] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 17:53:45,727 DEBUG [RS:3;jenkins-hbase4:35949] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 17:53:45,728 INFO [RS:3;jenkins-hbase4:35949] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,37439,1676310825610 with isa=jenkins-hbase4.apache.org/172.31.14.131:35949, startcode=1676310825652 2023-02-13 17:53:45,728 DEBUG [RS:3;jenkins-hbase4:35949] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 17:53:45,737 INFO [RS-EventLoopGroup-7-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42941, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.4 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 17:53:45,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37439] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:53:45,765 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x11b19bd3 to 127.0.0.1:52890 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:53:45,767 DEBUG [RS:3;jenkins-hbase4:35949] regionserver.HRegionServer(2829): Master is not running yet 2023-02-13 17:53:45,767 WARN [RS:3;jenkins-hbase4:35949] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-13 17:53:45,771 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7300d85e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:53:45,771 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 17:53:45,772 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-13 17:53:45,773 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 17:53:45,782 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/WALs/jenkins-hbase4.apache.org,38985,1676310817175 to hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/WALs/jenkins-hbase4.apache.org,38985,1676310817175-dead as it is dead 2023-02-13 17:53:45,784 INFO [master/jenkins-hbase4:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/WALs/jenkins-hbase4.apache.org,38985,1676310817175-dead/jenkins-hbase4.apache.org%2C38985%2C1676310817175.1676310819057 2023-02-13 17:53:45,789 INFO [master/jenkins-hbase4:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/WALs/jenkins-hbase4.apache.org,38985,1676310817175-dead/jenkins-hbase4.apache.org%2C38985%2C1676310817175.1676310819057 after 5ms 2023-02-13 17:53:45,790 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/WALs/jenkins-hbase4.apache.org,38985,1676310817175-dead/jenkins-hbase4.apache.org%2C38985%2C1676310817175.1676310819057 to hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C38985%2C1676310817175.1676310819057 2023-02-13 17:53:45,790 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/WALs/jenkins-hbase4.apache.org,38985,1676310817175-dead 2023-02-13 17:53:45,790 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/WALs/jenkins-hbase4.apache.org,37439,1676310825610 2023-02-13 17:53:45,793 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C37439%2C1676310825610, suffix=, logDir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/WALs/jenkins-hbase4.apache.org,37439,1676310825610, archiveDir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/oldWALs, maxLogs=10 2023-02-13 17:53:45,808 DEBUG [RS-EventLoopGroup-8-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33739,DS-07b12a43-1f57-44a8-a528-3a10ed60ab98,DISK] 2023-02-13 17:53:45,810 DEBUG [RS-EventLoopGroup-8-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33275,DS-6ebf9491-78fa-49e0-8c5b-82ee1005d4af,DISK] 2023-02-13 17:53:45,810 DEBUG [RS-EventLoopGroup-8-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45833,DS-0de4ecf6-a9b9-4f55-81ee-f85a32d928d3,DISK] 2023-02-13 17:53:45,813 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/WALs/jenkins-hbase4.apache.org,37439,1676310825610/jenkins-hbase4.apache.org%2C37439%2C1676310825610.1676310825793 2023-02-13 17:53:45,816 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33739,DS-07b12a43-1f57-44a8-a528-3a10ed60ab98,DISK], DatanodeInfoWithStorage[127.0.0.1:33275,DS-6ebf9491-78fa-49e0-8c5b-82ee1005d4af,DISK], DatanodeInfoWithStorage[127.0.0.1:45833,DS-0de4ecf6-a9b9-4f55-81ee-f85a32d928d3,DISK]] 2023-02-13 17:53:45,816 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-13 17:53:45,816 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:45,816 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 17:53:45,816 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 17:53:45,820 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-13 17:53:45,822 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-13 17:53:45,822 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-13 17:53:45,830 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(539): loaded hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/ac056f9f747149e8b3a96b94e282ec18 2023-02-13 17:53:45,831 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:45,833 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5178): Found 1 recovered edits file(s) under hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-02-13 17:53:45,833 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5275): Replaying edits from hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C38985%2C1676310817175.1676310819057 2023-02-13 17:53:45,852 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5463): Applied 0, skipped 160, firstSequenceIdInLog=3, maxSequenceIdInLog=143, path=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C38985%2C1676310817175.1676310819057 2023-02-13 17:53:45,853 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5085): Deleted recovered.edits file=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C38985%2C1676310817175.1676310819057 2023-02-13 17:53:45,858 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 17:53:45,860 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/143.seqid, newMaxSeqId=143, maxSeqId=1 2023-02-13 17:53:45,861 DEBUG [RS:2;jenkins-hbase4:34325] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 17:53:45,862 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=144; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60361914, jitterRate=-0.10053738951683044}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:53:45,862 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676310825816Initializing all the Stores at 1676310825818 (+2 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676310825818Cleaning up temporary data from old regions at 1676310825854 (+36 ms)Cleaning up detritus from prior splits at 1676310825854Region opened successfully at 1676310825862 (+8 ms) 2023-02-13 17:53:45,863 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-13 17:53:45,864 INFO [RS-EventLoopGroup-7-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35151, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 17:53:45,865 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-13 17:53:45,865 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-13 17:53:45,865 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=37439] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:545) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:53:45,865 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-13 17:53:45,866 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-13 17:53:45,867 INFO [RS:3;jenkins-hbase4:35949] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,37439,1676310825610 with isa=jenkins-hbase4.apache.org/172.31.14.131:35949, startcode=1676310825652 2023-02-13 17:53:45,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37439] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:53:45,869 DEBUG [RS:3;jenkins-hbase4:35949] regionserver.HRegionServer(2829): Master is not running yet 2023-02-13 17:53:45,869 WARN [RS:3;jenkins-hbase4:35949] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 200 ms and then retrying. 2023-02-13 17:53:45,880 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-13 17:53:45,880 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-13 17:53:45,880 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-13 17:53:45,880 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-13 17:53:45,881 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=restartTableOne 2023-02-13 17:53:45,881 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; CreateTableProcedure table=restartTableTwo 2023-02-13 17:53:45,881 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=15, state=SUCCESS; CreateTableProcedure table=restartTableThree 2023-02-13 17:53:45,882 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 15 msec 2023-02-13 17:53:45,882 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-13 17:53:45,882 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-13 17:53:45,883 DEBUG [RS:1;jenkins-hbase4:39175] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 17:53:45,883 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39175,1676310818498, table=hbase:meta, region=1588230740 2023-02-13 17:53:45,884 INFO [RS-EventLoopGroup-7-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34747, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 17:53:45,885 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=37439] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:545) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:53:45,885 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 3 possibly 'live' servers, and 0 'splitting'. 2023-02-13 17:53:45,886 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,34325,1676310818539 2023-02-13 17:53:45,887 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,39175,1676310818498 2023-02-13 17:53:45,888 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,35949,1676310818450 already deleted, retry=false 2023-02-13 17:53:45,888 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,35949,1676310818450 on jenkins-hbase4.apache.org,37439,1676310825610 2023-02-13 17:53:45,896 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=18, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,35949,1676310818450, splitWal=true, meta=false 2023-02-13 17:53:45,896 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1736): Scheduled ServerCrashProcedure pid=18 for jenkins-hbase4.apache.org,35949,1676310818450 (carryingMeta=false) jenkins-hbase4.apache.org,35949,1676310818450/CRASHED/regionCount=0/lock=java.util.concurrent.locks.ReentrantReadWriteLock@2143d726[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-02-13 17:53:45,907 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-13 17:53:45,907 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 17:53:45,907 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/balancer 2023-02-13 17:53:45,908 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-13 17:53:45,908 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-13 17:53:45,909 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-13 17:53:45,909 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-13 17:53:45,910 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-13 17:53:45,910 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,37439,1676310825610, sessionid=0x1004ed10fed0009, setting cluster-up flag (Was=true) 2023-02-13 17:53:45,912 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-13 17:53:45,913 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,37439,1676310825610 2023-02-13 17:53:45,915 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-13 17:53:45,916 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,37439,1676310825610 2023-02-13 17:53:45,916 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/.hbase-snapshot/.tmp 2023-02-13 17:53:45,920 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 17:53:45,920 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 17:53:45,920 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 17:53:45,920 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 17:53:45,920 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-13 17:53:45,921 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:45,921 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 17:53:45,921 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:45,924 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676310855923 2023-02-13 17:53:45,924 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-13 17:53:45,924 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-13 17:53:45,924 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-13 17:53:45,924 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-13 17:53:45,925 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-13 17:53:45,925 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-13 17:53:45,925 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:45,926 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-13 17:53:45,926 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-13 17:53:45,926 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-13 17:53:45,926 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-13 17:53:45,927 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-13 17:53:45,927 DEBUG [PEWorker-1] master.DeadServer(103): Processing jenkins-hbase4.apache.org,35949,1676310818450; numProcessing=1 2023-02-13 17:53:45,927 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676310825927,5,FailOnTimeoutGroup] 2023-02-13 17:53:45,928 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676310825928,5,FailOnTimeoutGroup] 2023-02-13 17:53:45,928 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:45,928 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-13 17:53:45,928 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:45,928 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:45,928 INFO [PEWorker-1] procedure.ServerCrashProcedure(161): Start pid=18, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,35949,1676310818450, splitWal=true, meta=false 2023-02-13 17:53:45,928 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676310825643, completionTime=-1 2023-02-13 17:53:45,928 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(801): Waiting on regionserver count=2; waited=0ms, expecting min=3 server(s), max=3 server(s), timeout=4500ms, lastChange=0ms 2023-02-13 17:53:45,958 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 17:53:46,025 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-13 17:53:46,027 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-13 17:53:46,027 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'restartTableThree' 2023-02-13 17:53:46,070 INFO [RS:3;jenkins-hbase4:35949] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,37439,1676310825610 with isa=jenkins-hbase4.apache.org/172.31.14.131:35949, startcode=1676310825652 2023-02-13 17:53:46,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37439] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,35949,1676310825652 2023-02-13 17:53:46,072 DEBUG [RS:3;jenkins-hbase4:35949] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2 2023-02-13 17:53:46,072 DEBUG [RS:3;jenkins-hbase4:35949] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:42429 2023-02-13 17:53:46,072 DEBUG [RS:3;jenkins-hbase4:35949] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 17:53:46,075 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:34325-0x1004ed10fed0003, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:46,075 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:46,075 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:46,075 DEBUG [RS:3;jenkins-hbase4:35949] zookeeper.ZKUtil(162): regionserver:35949-0x1004ed10fed000a, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35949,1676310825652 2023-02-13 17:53:46,076 WARN [RS:3;jenkins-hbase4:35949] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 17:53:46,076 INFO [RS:3;jenkins-hbase4:35949] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 17:53:46,076 DEBUG [RS:3;jenkins-hbase4:35949] regionserver.HRegionServer(1947): logDir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/WALs/jenkins-hbase4.apache.org,35949,1676310825652 2023-02-13 17:53:46,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34325,1676310818539 2023-02-13 17:53:46,076 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,35949,1676310825652] 2023-02-13 17:53:46,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34325-0x1004ed10fed0003, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34325,1676310818539 2023-02-13 17:53:46,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35949,1676310825652 2023-02-13 17:53:46,079 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=151ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-13 17:53:46,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34325-0x1004ed10fed0003, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35949,1676310825652 2023-02-13 17:53:46,079 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-13 17:53:46,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39175,1676310818498 2023-02-13 17:53:46,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34325-0x1004ed10fed0003, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39175,1676310818498 2023-02-13 17:53:46,082 DEBUG [RS:3;jenkins-hbase4:35949] zookeeper.ZKUtil(162): regionserver:35949-0x1004ed10fed000a, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34325,1676310818539 2023-02-13 17:53:46,082 DEBUG [hconnection-0x35ed1ec1-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:53:46,083 DEBUG [RS:3;jenkins-hbase4:35949] zookeeper.ZKUtil(162): regionserver:35949-0x1004ed10fed000a, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35949,1676310825652 2023-02-13 17:53:46,083 DEBUG [RS:3;jenkins-hbase4:35949] zookeeper.ZKUtil(162): regionserver:35949-0x1004ed10fed000a, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39175,1676310818498 2023-02-13 17:53:46,085 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48150, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=ClientService 2023-02-13 17:53:46,085 DEBUG [RS:3;jenkins-hbase4:35949] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 17:53:46,086 INFO [RS:3;jenkins-hbase4:35949] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 17:53:46,088 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=673bbd347d5066108d814a6b9fc9e57c, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39175,1676310818498, regionLocation=jenkins-hbase4.apache.org,39175,1676310818498, openSeqNum=2 2023-02-13 17:53:46,088 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=f3a07b2ad33c42a90eb09beaac5e4045, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,35949,1676310818450, regionLocation=jenkins-hbase4.apache.org,35949,1676310818450, openSeqNum=2 2023-02-13 17:53:46,090 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=f1791fa38e7b82b917ae39ff48365a4d, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39175,1676310818498, regionLocation=jenkins-hbase4.apache.org,39175,1676310818498, openSeqNum=2 2023-02-13 17:53:46,090 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=dc2f11e16ffd6ee6c6b4b51c20c090b8, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,35949,1676310818450, regionLocation=jenkins-hbase4.apache.org,35949,1676310818450, openSeqNum=2 2023-02-13 17:53:46,090 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-13 17:53:46,090 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676310886090 2023-02-13 17:53:46,090 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676310946090 2023-02-13 17:53:46,091 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 11 msec 2023-02-13 17:53:46,092 INFO [RS:3;jenkins-hbase4:35949] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 17:53:46,102 INFO [RS:3;jenkins-hbase4:35949] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 17:53:46,102 INFO [RS:3;jenkins-hbase4:35949] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:46,104 INFO [RS:3;jenkins-hbase4:35949] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 17:53:46,106 INFO [RS:3;jenkins-hbase4:35949] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:46,106 DEBUG [RS:3;jenkins-hbase4:35949] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:35949, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:46,106 DEBUG [RS:3;jenkins-hbase4:35949] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:35949, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:46,106 DEBUG [RS:3;jenkins-hbase4:35949] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:35949, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:46,106 DEBUG [RS:3;jenkins-hbase4:35949] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:35949, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:46,106 DEBUG [RS:3;jenkins-hbase4:35949] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:35949, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:46,106 DEBUG [RS:3;jenkins-hbase4:35949] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:35949, corePoolSize=2, maxPoolSize=2 2023-02-13 17:53:46,106 DEBUG [RS:3;jenkins-hbase4:35949] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:35949, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:46,106 DEBUG [RS:3;jenkins-hbase4:35949] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:35949, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:46,107 DEBUG [RS:3;jenkins-hbase4:35949] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:35949, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:46,107 DEBUG [RS:3;jenkins-hbase4:35949] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:35949, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:46,108 INFO [RS:3;jenkins-hbase4:35949] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:46,108 INFO [RS:3;jenkins-hbase4:35949] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:46,108 INFO [RS:3;jenkins-hbase4:35949] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:46,123 INFO [RS:3;jenkins-hbase4:35949] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 17:53:46,123 INFO [RS:3;jenkins-hbase4:35949] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35949,1676310825652-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:46,128 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37439,1676310825610-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:46,128 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37439,1676310825610-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:46,128 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37439,1676310825610-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:46,128 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:37439, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:46,128 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:46,128 INFO [PEWorker-2] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,35949,1676310818450 had 2 regions 2023-02-13 17:53:46,129 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-02-13 17:53:46,131 INFO [PEWorker-2] procedure.ServerCrashProcedure(300): Splitting WALs pid=18, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,35949,1676310818450, splitWal=true, meta=false, isMeta: false 2023-02-13 17:53:46,134 DEBUG [PEWorker-2] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/WALs/jenkins-hbase4.apache.org,35949,1676310818450-splitting 2023-02-13 17:53:46,136 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/WALs/jenkins-hbase4.apache.org,35949,1676310818450-splitting dir is empty, no logs to split. 2023-02-13 17:53:46,136 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,35949,1676310818450 WAL count=0, meta=false 2023-02-13 17:53:46,145 WARN [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(172): unknown_server=jenkins-hbase4.apache.org,35949,1676310818450/restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045., unknown_server=jenkins-hbase4.apache.org,35949,1676310818450/restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8. 2023-02-13 17:53:46,146 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/WALs/jenkins-hbase4.apache.org,35949,1676310818450-splitting dir is empty, no logs to split. 2023-02-13 17:53:46,146 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,35949,1676310818450 WAL count=0, meta=false 2023-02-13 17:53:46,146 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,35949,1676310818450 WAL splitting is done? wals=0, meta=false 2023-02-13 17:53:46,149 INFO [PEWorker-2] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase4.apache.org,35949,1676310818450 failed, ignore...File hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/WALs/jenkins-hbase4.apache.org,35949,1676310818450-splitting does not exist. 2023-02-13 17:53:46,152 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=dc2f11e16ffd6ee6c6b4b51c20c090b8, ASSIGN}, {pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=f3a07b2ad33c42a90eb09beaac5e4045, ASSIGN}] 2023-02-13 17:53:46,160 INFO [RS:3;jenkins-hbase4:35949] regionserver.Replication(203): jenkins-hbase4.apache.org,35949,1676310825652 started 2023-02-13 17:53:46,160 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=f3a07b2ad33c42a90eb09beaac5e4045, ASSIGN 2023-02-13 17:53:46,161 INFO [RS:3;jenkins-hbase4:35949] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,35949,1676310825652, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:35949, sessionid=0x1004ed10fed000a 2023-02-13 17:53:46,168 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=dc2f11e16ffd6ee6c6b4b51c20c090b8, ASSIGN 2023-02-13 17:53:46,170 DEBUG [RS:3;jenkins-hbase4:35949] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 17:53:46,170 DEBUG [RS:3;jenkins-hbase4:35949] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,35949,1676310825652 2023-02-13 17:53:46,170 DEBUG [RS:3;jenkins-hbase4:35949] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35949,1676310825652' 2023-02-13 17:53:46,170 DEBUG [RS:3;jenkins-hbase4:35949] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 17:53:46,170 INFO [Listener at localhost/43465] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:46,171 DEBUG [RS:3;jenkins-hbase4:35949] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 17:53:46,171 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,35949,1676310818450 2023-02-13 17:53:46,171 DEBUG [RS:3;jenkins-hbase4:35949] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 17:53:46,172 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableOne, region=f3a07b2ad33c42a90eb09beaac5e4045, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,35949,1676310818450; forceNewPlan=false, retain=true 2023-02-13 17:53:46,172 DEBUG [RS:3;jenkins-hbase4:35949] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 17:53:46,172 DEBUG [RS:3;jenkins-hbase4:35949] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,35949,1676310825652 2023-02-13 17:53:46,172 DEBUG [RS:3;jenkins-hbase4:35949] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35949,1676310825652' 2023-02-13 17:53:46,172 DEBUG [RS:3;jenkins-hbase4:35949] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 17:53:46,172 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,35949,1676310818450 2023-02-13 17:53:46,172 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableTwo, region=dc2f11e16ffd6ee6c6b4b51c20c090b8, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,35949,1676310818450; forceNewPlan=false, retain=true 2023-02-13 17:53:46,174 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-13 17:53:46,176 DEBUG [RS:3;jenkins-hbase4:35949] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 17:53:46,179 DEBUG [RS:3;jenkins-hbase4:35949] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 17:53:46,179 INFO [RS:3;jenkins-hbase4:35949] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 17:53:46,179 INFO [RS:3;jenkins-hbase4:35949] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 17:53:46,179 WARN [RPCClient-NioEventLoopGroup-6-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:38985 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:38985 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 17:53:46,180 WARN [Listener at localhost/43465] client.ConnectionImplementation(411): Checking master connection java.net.ConnectException: Call to address=jenkins-hbase4.apache.org/172.31.14.131:38985 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:38985 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:186) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:386) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:417) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:413) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:99) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:398) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:368) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1428) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:396) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:913) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:195) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$300(NettyRpcConnection.java:76) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:296) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:287) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:38985 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 17:53:46,181 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 0.531sec 2023-02-13 17:53:46,180 DEBUG [RPCClient-NioEventLoopGroup-6-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:38985 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:38985 2023-02-13 17:53:46,180 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-13 17:53:46,181 DEBUG [Listener at localhost/43465] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:38985 this server is in the failed servers list 2023-02-13 17:53:46,182 WARN [Listener at localhost/43465] client.ConnectionImplementation(411): Checking master connection org.apache.hadoop.hbase.ipc.FailedServerException: Call to address=jenkins-hbase4.apache.org/172.31.14.131:38985 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: jenkins-hbase4.apache.org/172.31.14.131:38985 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:236) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:386) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:417) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:413) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:430) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:326) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:582) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1174) at org.apache.hadoop.hbase.client.ConnectionImplementation.lambda$masterConnectionStateSupplier$1(ConnectionImplementation.java:404) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1945) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1383) at org.apache.hadoop.hbase.client.ConnectionImplementation.getMaster(ConnectionImplementation.java:1373) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:57) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:102) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:2954) at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:542) at org.apache.hadoop.hbase.client.HTable.getDescriptor(HTable.java:228) at org.apache.hadoop.hbase.HBaseTestingUtility$9.evaluate(HBaseTestingUtility.java:3983) at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:183) at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:134) at org.apache.hadoop.hbase.HBaseCommonTestingUtility.waitFor(HBaseCommonTestingUtility.java:234) at org.apache.hadoop.hbase.HBaseTestingUtility.waitTableAvailable(HBaseTestingUtility.java:3136) at org.apache.hadoop.hbase.HBaseTestingUtility.waitTableAvailable(HBaseTestingUtility.java:3121) at org.apache.hadoop.hbase.master.TestRetainAssignmentOnRestart.testRetainAssignmentOnSingleRSRestart(TestRetainAssignmentOnRestart.java:176) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: jenkins-hbase4.apache.org/172.31.14.131:38985 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:350) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:427) ... 46 more 2023-02-13 17:53:46,182 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-13 17:53:46,182 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-13 17:53:46,182 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-13 17:53:46,185 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37439,1676310825610-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-13 17:53:46,185 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37439,1676310825610-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-13 17:53:46,186 DEBUG [Listener at localhost/43465] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-13 17:53:46,192 INFO [RS-EventLoopGroup-7-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56936, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-13 17:53:46,196 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-13 17:53:46,207 DEBUG [hconnection-0x7dff9cd5-shared-pool-6] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:53:46,209 INFO [RS-EventLoopGroup-8-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56978, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:53:46,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35949] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Scan size: 155 connection: 172.31.14.131:56978 deadline: 1676310886208, exception=org.apache.hadoop.hbase.NotServingRegionException: restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045. is not online on jenkins-hbase4.apache.org,35949,1676310825652 2023-02-13 17:53:46,282 INFO [RS:3;jenkins-hbase4:35949] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35949%2C1676310825652, suffix=, logDir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/WALs/jenkins-hbase4.apache.org,35949,1676310825652, archiveDir=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/oldWALs, maxLogs=32 2023-02-13 17:53:46,309 DEBUG [RS-EventLoopGroup-8-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45833,DS-0de4ecf6-a9b9-4f55-81ee-f85a32d928d3,DISK] 2023-02-13 17:53:46,309 DEBUG [RS-EventLoopGroup-8-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33275,DS-6ebf9491-78fa-49e0-8c5b-82ee1005d4af,DISK] 2023-02-13 17:53:46,309 DEBUG [RS-EventLoopGroup-8-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33739,DS-07b12a43-1f57-44a8-a528-3a10ed60ab98,DISK] 2023-02-13 17:53:46,353 INFO [RS:3;jenkins-hbase4:35949] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/WALs/jenkins-hbase4.apache.org,35949,1676310825652/jenkins-hbase4.apache.org%2C35949%2C1676310825652.1676310826284 2023-02-13 17:53:46,354 INFO [jenkins-hbase4:37439] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-02-13 17:53:46,354 DEBUG [RS:3;jenkins-hbase4:35949] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45833,DS-0de4ecf6-a9b9-4f55-81ee-f85a32d928d3,DISK], DatanodeInfoWithStorage[127.0.0.1:33275,DS-6ebf9491-78fa-49e0-8c5b-82ee1005d4af,DISK], DatanodeInfoWithStorage[127.0.0.1:33739,DS-07b12a43-1f57-44a8-a528-3a10ed60ab98,DISK]] 2023-02-13 17:53:46,357 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=f3a07b2ad33c42a90eb09beaac5e4045, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35949,1676310825652 2023-02-13 17:53:46,357 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=dc2f11e16ffd6ee6c6b4b51c20c090b8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35949,1676310825652 2023-02-13 17:53:46,357 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676310826357"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676310826357"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676310826357"}]},"ts":"1676310826357"} 2023-02-13 17:53:46,357 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676310826357"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676310826357"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676310826357"}]},"ts":"1676310826357"} 2023-02-13 17:53:46,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35949] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Scan size: 155 connection: 172.31.14.131:56978 deadline: 1676310886207, exception=org.apache.hadoop.hbase.NotServingRegionException: restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045. is not online on jenkins-hbase4.apache.org,35949,1676310825652 2023-02-13 17:53:46,361 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE; OpenRegionProcedure f3a07b2ad33c42a90eb09beaac5e4045, server=jenkins-hbase4.apache.org,35949,1676310825652}] 2023-02-13 17:53:46,362 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=19, state=RUNNABLE; OpenRegionProcedure dc2f11e16ffd6ee6c6b4b51c20c090b8, server=jenkins-hbase4.apache.org,35949,1676310825652}] 2023-02-13 17:53:46,519 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,35949,1676310825652 2023-02-13 17:53:46,519 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 17:53:46,521 INFO [RS-EventLoopGroup-8-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56980, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=AdminService 2023-02-13 17:53:46,525 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] handler.AssignRegionHandler(128): Open restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8. 2023-02-13 17:53:46,526 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(7850): Opening region: {ENCODED => dc2f11e16ffd6ee6c6b4b51c20c090b8, NAME => 'restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8.', STARTKEY => '', ENDKEY => ''} 2023-02-13 17:53:46,526 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableTwo dc2f11e16ffd6ee6c6b4b51c20c090b8 2023-02-13 17:53:46,526 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:46,526 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(7890): checking encryption for dc2f11e16ffd6ee6c6b4b51c20c090b8 2023-02-13 17:53:46,526 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(7893): checking classloading for dc2f11e16ffd6ee6c6b4b51c20c090b8 2023-02-13 17:53:46,528 INFO [StoreOpener-dc2f11e16ffd6ee6c6b4b51c20c090b8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region dc2f11e16ffd6ee6c6b4b51c20c090b8 2023-02-13 17:53:46,529 DEBUG [StoreOpener-dc2f11e16ffd6ee6c6b4b51c20c090b8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableTwo/dc2f11e16ffd6ee6c6b4b51c20c090b8/family 2023-02-13 17:53:46,529 DEBUG [StoreOpener-dc2f11e16ffd6ee6c6b4b51c20c090b8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableTwo/dc2f11e16ffd6ee6c6b4b51c20c090b8/family 2023-02-13 17:53:46,530 INFO [StoreOpener-dc2f11e16ffd6ee6c6b4b51c20c090b8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region dc2f11e16ffd6ee6c6b4b51c20c090b8 columnFamilyName family 2023-02-13 17:53:46,530 INFO [StoreOpener-dc2f11e16ffd6ee6c6b4b51c20c090b8-1] regionserver.HStore(310): Store=dc2f11e16ffd6ee6c6b4b51c20c090b8/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:46,531 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableTwo/dc2f11e16ffd6ee6c6b4b51c20c090b8 2023-02-13 17:53:46,533 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableTwo/dc2f11e16ffd6ee6c6b4b51c20c090b8 2023-02-13 17:53:46,537 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(1054): writing seq id for dc2f11e16ffd6ee6c6b4b51c20c090b8 2023-02-13 17:53:46,538 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(1071): Opened dc2f11e16ffd6ee6c6b4b51c20c090b8; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63349636, jitterRate=-0.05601686239242554}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:53:46,538 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(964): Region open journal for dc2f11e16ffd6ee6c6b4b51c20c090b8: Running coprocessor pre-open hook at 1676310826526Writing region info on filesystem at 1676310826526Initializing all the Stores at 1676310826527 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676310826527Cleaning up temporary data from old regions at 1676310826534 (+7 ms)Cleaning up detritus from prior splits at 1676310826534Running coprocessor post-open hooks at 1676310826538 (+4 ms)Region opened successfully at 1676310826538 2023-02-13 17:53:46,539 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8., pid=22, masterSystemTime=1676310826519 2023-02-13 17:53:46,543 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8. 2023-02-13 17:53:46,544 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] handler.AssignRegionHandler(156): Opened restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8. 2023-02-13 17:53:46,544 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] handler.AssignRegionHandler(128): Open restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045. 2023-02-13 17:53:46,544 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(7850): Opening region: {ENCODED => f3a07b2ad33c42a90eb09beaac5e4045, NAME => 'restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045.', STARTKEY => '', ENDKEY => ''} 2023-02-13 17:53:46,544 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableOne f3a07b2ad33c42a90eb09beaac5e4045 2023-02-13 17:53:46,544 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:46,544 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(7890): checking encryption for f3a07b2ad33c42a90eb09beaac5e4045 2023-02-13 17:53:46,544 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(7893): checking classloading for f3a07b2ad33c42a90eb09beaac5e4045 2023-02-13 17:53:46,545 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=dc2f11e16ffd6ee6c6b4b51c20c090b8, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35949,1676310825652 2023-02-13 17:53:46,545 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676310826544"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676310826544"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676310826544"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676310826544"}]},"ts":"1676310826544"} 2023-02-13 17:53:46,546 INFO [StoreOpener-f3a07b2ad33c42a90eb09beaac5e4045-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region f3a07b2ad33c42a90eb09beaac5e4045 2023-02-13 17:53:46,548 DEBUG [StoreOpener-f3a07b2ad33c42a90eb09beaac5e4045-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableOne/f3a07b2ad33c42a90eb09beaac5e4045/family 2023-02-13 17:53:46,548 DEBUG [StoreOpener-f3a07b2ad33c42a90eb09beaac5e4045-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableOne/f3a07b2ad33c42a90eb09beaac5e4045/family 2023-02-13 17:53:46,548 INFO [StoreOpener-f3a07b2ad33c42a90eb09beaac5e4045-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f3a07b2ad33c42a90eb09beaac5e4045 columnFamilyName family 2023-02-13 17:53:46,549 INFO [StoreOpener-f3a07b2ad33c42a90eb09beaac5e4045-1] regionserver.HStore(310): Store=f3a07b2ad33c42a90eb09beaac5e4045/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:46,550 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=19 2023-02-13 17:53:46,550 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=19, state=SUCCESS; OpenRegionProcedure dc2f11e16ffd6ee6c6b4b51c20c090b8, server=jenkins-hbase4.apache.org,35949,1676310825652 in 185 msec 2023-02-13 17:53:46,551 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableOne/f3a07b2ad33c42a90eb09beaac5e4045 2023-02-13 17:53:46,553 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableOne/f3a07b2ad33c42a90eb09beaac5e4045 2023-02-13 17:53:46,554 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=restartTableTwo, region=dc2f11e16ffd6ee6c6b4b51c20c090b8, ASSIGN in 398 msec 2023-02-13 17:53:46,556 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(1054): writing seq id for f3a07b2ad33c42a90eb09beaac5e4045 2023-02-13 17:53:46,558 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(1071): Opened f3a07b2ad33c42a90eb09beaac5e4045; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73717960, jitterRate=0.09848320484161377}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:53:46,558 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(964): Region open journal for f3a07b2ad33c42a90eb09beaac5e4045: Running coprocessor pre-open hook at 1676310826545Writing region info on filesystem at 1676310826545Initializing all the Stores at 1676310826546 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676310826546Cleaning up temporary data from old regions at 1676310826554 (+8 ms)Cleaning up detritus from prior splits at 1676310826554Running coprocessor post-open hooks at 1676310826558 (+4 ms)Region opened successfully at 1676310826558 2023-02-13 17:53:46,558 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045., pid=21, masterSystemTime=1676310826519 2023-02-13 17:53:46,560 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045. 2023-02-13 17:53:46,561 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:35949-0] handler.AssignRegionHandler(156): Opened restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045. 2023-02-13 17:53:46,561 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=f3a07b2ad33c42a90eb09beaac5e4045, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35949,1676310825652 2023-02-13 17:53:46,562 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676310826561"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676310826561"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676310826561"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676310826561"}]},"ts":"1676310826561"} 2023-02-13 17:53:46,567 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-02-13 17:53:46,567 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; OpenRegionProcedure f3a07b2ad33c42a90eb09beaac5e4045, server=jenkins-hbase4.apache.org,35949,1676310825652 in 203 msec 2023-02-13 17:53:46,568 INFO [Listener at localhost/43465] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:46,571 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=18 2023-02-13 17:53:46,571 INFO [PEWorker-4] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,35949,1676310818450 after splitting done 2023-02-13 17:53:46,571 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=restartTableOne, region=f3a07b2ad33c42a90eb09beaac5e4045, ASSIGN in 415 msec 2023-02-13 17:53:46,571 DEBUG [PEWorker-4] master.DeadServer(114): Removed jenkins-hbase4.apache.org,35949,1676310818450 from processing; numProcessing=0 2023-02-13 17:53:46,574 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,35949,1676310818450, splitWal=true, meta=false in 681 msec 2023-02-13 17:53:46,579 INFO [Listener at localhost/43465] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:46,587 INFO [Listener at localhost/43465] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:46,587 INFO [Listener at localhost/43465] master.SnapshotOfRegionAssignmentFromMeta(102): Start to scan the hbase:meta for the current region assignment snappshot 2023-02-13 17:53:46,591 INFO [Listener at localhost/43465] master.SnapshotOfRegionAssignmentFromMeta(164): Finished to scan the hbase:meta for the current region assignmentsnapshot 2023-02-13 17:53:46,591 INFO [Listener at localhost/43465] master.TestRetainAssignmentOnRestart(187): Key={ENCODED => dc2f11e16ffd6ee6c6b4b51c20c090b8, NAME => 'restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,35949,1676310818450, currentServer=jenkins-hbase4.apache.org,35949,1676310825652 2023-02-13 17:53:46,592 INFO [Listener at localhost/43465] master.TestRetainAssignmentOnRestart(187): Key={ENCODED => f3a07b2ad33c42a90eb09beaac5e4045, NAME => 'restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,35949,1676310818450, currentServer=jenkins-hbase4.apache.org,35949,1676310825652 2023-02-13 17:53:46,592 INFO [Listener at localhost/43465] master.TestRetainAssignmentOnRestart(187): Key={ENCODED => 673bbd347d5066108d814a6b9fc9e57c, NAME => 'hbase:namespace,,1676310820635.673bbd347d5066108d814a6b9fc9e57c.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,39175,1676310818498, currentServer=jenkins-hbase4.apache.org,39175,1676310818498 2023-02-13 17:53:46,592 INFO [Listener at localhost/43465] master.TestRetainAssignmentOnRestart(187): Key={ENCODED => f1791fa38e7b82b917ae39ff48365a4d, NAME => 'restartTableThree,,1676310822711.f1791fa38e7b82b917ae39ff48365a4d.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,39175,1676310818498, currentServer=jenkins-hbase4.apache.org,39175,1676310818498 2023-02-13 17:53:46,592 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-13 17:53:46,592 INFO [Listener at localhost/43465] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-13 17:53:46,592 DEBUG [Listener at localhost/43465] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x26fb0c0f to 127.0.0.1:52890 2023-02-13 17:53:46,593 DEBUG [Listener at localhost/43465] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:46,593 DEBUG [Listener at localhost/43465] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-13 17:53:46,593 DEBUG [Listener at localhost/43465] util.JVMClusterUtil(257): Found active master hash=603828848, stopped=false 2023-02-13 17:53:46,593 INFO [Listener at localhost/43465] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,37439,1676310825610 2023-02-13 17:53:46,595 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 17:53:46,595 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:34325-0x1004ed10fed0003, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 17:53:46,595 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 17:53:46,595 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:35949-0x1004ed10fed000a, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 17:53:46,595 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:53:46,595 INFO [Listener at localhost/43465] procedure2.ProcedureExecutor(629): Stopping 2023-02-13 17:53:46,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 17:53:46,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34325-0x1004ed10fed0003, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 17:53:46,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 17:53:46,597 DEBUG [Listener at localhost/43465] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x11b19bd3 to 127.0.0.1:52890 2023-02-13 17:53:46,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35949-0x1004ed10fed000a, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 17:53:46,597 DEBUG [Listener at localhost/43465] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:46,598 INFO [Listener at localhost/43465] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,39175,1676310818498' ***** 2023-02-13 17:53:46,598 INFO [Listener at localhost/43465] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 17:53:46,598 INFO [Listener at localhost/43465] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,34325,1676310818539' ***** 2023-02-13 17:53:46,598 INFO [Listener at localhost/43465] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 17:53:46,598 INFO [RS:1;jenkins-hbase4:39175] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 17:53:46,598 INFO [Listener at localhost/43465] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,35949,1676310825652' ***** 2023-02-13 17:53:46,598 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 17:53:46,598 INFO [Listener at localhost/43465] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 17:53:46,598 INFO [RS:1;jenkins-hbase4:39175] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 17:53:46,598 INFO [RS:2;jenkins-hbase4:34325] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 17:53:46,599 INFO [RS:3;jenkins-hbase4:35949] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 17:53:46,599 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 17:53:46,599 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 17:53:46,599 INFO [RS:1;jenkins-hbase4:39175] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 17:53:46,599 INFO [RS:3;jenkins-hbase4:35949] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 17:53:46,600 INFO [RS:1;jenkins-hbase4:39175] regionserver.HRegionServer(3304): Received CLOSE for f1791fa38e7b82b917ae39ff48365a4d 2023-02-13 17:53:46,599 INFO [RS:2;jenkins-hbase4:34325] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 17:53:46,600 INFO [RS:3;jenkins-hbase4:35949] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 17:53:46,600 INFO [RS:2;jenkins-hbase4:34325] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 17:53:46,600 INFO [RS:3;jenkins-hbase4:35949] regionserver.HRegionServer(3304): Received CLOSE for f3a07b2ad33c42a90eb09beaac5e4045 2023-02-13 17:53:46,600 INFO [RS:2;jenkins-hbase4:34325] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,34325,1676310818539 2023-02-13 17:53:46,600 INFO [RS:1;jenkins-hbase4:39175] regionserver.HRegionServer(3304): Received CLOSE for 673bbd347d5066108d814a6b9fc9e57c 2023-02-13 17:53:46,601 DEBUG [RS:2;jenkins-hbase4:34325] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50ab9e36 to 127.0.0.1:52890 2023-02-13 17:53:46,601 INFO [RS:3;jenkins-hbase4:35949] regionserver.HRegionServer(3304): Received CLOSE for dc2f11e16ffd6ee6c6b4b51c20c090b8 2023-02-13 17:53:46,601 INFO [RS:1;jenkins-hbase4:39175] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,39175,1676310818498 2023-02-13 17:53:46,601 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f1791fa38e7b82b917ae39ff48365a4d, disabling compactions & flushes 2023-02-13 17:53:46,601 DEBUG [RS:1;jenkins-hbase4:39175] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x72cc9fb0 to 127.0.0.1:52890 2023-02-13 17:53:46,601 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(1603): Closing f3a07b2ad33c42a90eb09beaac5e4045, disabling compactions & flushes 2023-02-13 17:53:46,601 INFO [RS:3;jenkins-hbase4:35949] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,35949,1676310825652 2023-02-13 17:53:46,601 DEBUG [RS:2;jenkins-hbase4:34325] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:46,602 DEBUG [RS:3;jenkins-hbase4:35949] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x64cbde09 to 127.0.0.1:52890 2023-02-13 17:53:46,602 INFO [RS:2;jenkins-hbase4:34325] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,34325,1676310818539; all regions closed. 2023-02-13 17:53:46,602 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045. 2023-02-13 17:53:46,602 DEBUG [RS:1;jenkins-hbase4:39175] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:46,602 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region restartTableThree,,1676310822711.f1791fa38e7b82b917ae39ff48365a4d. 2023-02-13 17:53:46,602 INFO [RS:1;jenkins-hbase4:39175] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 17:53:46,602 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045. 2023-02-13 17:53:46,602 DEBUG [RS:3;jenkins-hbase4:35949] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:46,602 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045. after waiting 0 ms 2023-02-13 17:53:46,602 INFO [RS:1;jenkins-hbase4:39175] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 17:53:46,602 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableThree,,1676310822711.f1791fa38e7b82b917ae39ff48365a4d. 2023-02-13 17:53:46,602 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableThree,,1676310822711.f1791fa38e7b82b917ae39ff48365a4d. after waiting 0 ms 2023-02-13 17:53:46,602 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableThree,,1676310822711.f1791fa38e7b82b917ae39ff48365a4d. 2023-02-13 17:53:46,602 INFO [RS:1;jenkins-hbase4:39175] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 17:53:46,602 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045. 2023-02-13 17:53:46,602 INFO [RS:3;jenkins-hbase4:35949] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-13 17:53:46,603 INFO [RS:1;jenkins-hbase4:39175] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-13 17:53:46,603 DEBUG [RS:3;jenkins-hbase4:35949] regionserver.HRegionServer(1479): Online Regions={f3a07b2ad33c42a90eb09beaac5e4045=restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045., dc2f11e16ffd6ee6c6b4b51c20c090b8=restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8.} 2023-02-13 17:53:46,603 DEBUG [RS:3;jenkins-hbase4:35949] regionserver.HRegionServer(1505): Waiting on dc2f11e16ffd6ee6c6b4b51c20c090b8, f3a07b2ad33c42a90eb09beaac5e4045 2023-02-13 17:53:46,603 INFO [RS:1;jenkins-hbase4:39175] regionserver.HRegionServer(1475): Waiting on 3 regions to close 2023-02-13 17:53:46,604 DEBUG [RS:1;jenkins-hbase4:39175] regionserver.HRegionServer(1479): Online Regions={f1791fa38e7b82b917ae39ff48365a4d=restartTableThree,,1676310822711.f1791fa38e7b82b917ae39ff48365a4d., 1588230740=hbase:meta,,1.1588230740, 673bbd347d5066108d814a6b9fc9e57c=hbase:namespace,,1676310820635.673bbd347d5066108d814a6b9fc9e57c.} 2023-02-13 17:53:46,604 DEBUG [RS:1;jenkins-hbase4:39175] regionserver.HRegionServer(1505): Waiting on 1588230740, 673bbd347d5066108d814a6b9fc9e57c, f1791fa38e7b82b917ae39ff48365a4d 2023-02-13 17:53:46,606 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 17:53:46,606 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 17:53:46,606 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 17:53:46,606 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 17:53:46,606 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 17:53:46,606 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=7.04 KB heapSize=12.50 KB 2023-02-13 17:53:46,609 INFO [regionserver/jenkins-hbase4:35949.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 17:53:46,611 DEBUG [RS:2;jenkins-hbase4:34325] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/oldWALs 2023-02-13 17:53:46,611 INFO [RS:2;jenkins-hbase4:34325] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34325%2C1676310818539:(num 1676310820086) 2023-02-13 17:53:46,611 DEBUG [RS:2;jenkins-hbase4:34325] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:46,611 INFO [RS:2;jenkins-hbase4:34325] regionserver.LeaseManager(133): Closed leases 2023-02-13 17:53:46,611 INFO [RS:2;jenkins-hbase4:34325] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-13 17:53:46,611 INFO [RS:2;jenkins-hbase4:34325] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 17:53:46,611 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 17:53:46,611 INFO [RS:2;jenkins-hbase4:34325] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 17:53:46,611 INFO [RS:2;jenkins-hbase4:34325] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 17:53:46,612 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableThree/f1791fa38e7b82b917ae39ff48365a4d/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 17:53:46,612 INFO [RS:2;jenkins-hbase4:34325] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34325 2023-02-13 17:53:46,615 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:35949-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableOne/f3a07b2ad33c42a90eb09beaac5e4045/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-13 17:53:46,617 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:35949-0x1004ed10fed000a, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34325,1676310818539 2023-02-13 17:53:46,617 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:34325-0x1004ed10fed0003, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34325,1676310818539 2023-02-13 17:53:46,617 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34325,1676310818539 2023-02-13 17:53:46,617 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed restartTableThree,,1676310822711.f1791fa38e7b82b917ae39ff48365a4d. 2023-02-13 17:53:46,617 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:34325-0x1004ed10fed0003, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:46,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f1791fa38e7b82b917ae39ff48365a4d: Waiting for close lock at 1676310826600Running coprocessor pre-close hooks at 1676310826601 (+1 ms)Disabling compacts and flushes for region at 1676310826601Disabling writes for close at 1676310826602 (+1 ms)Writing region close event to WAL at 1676310826603 (+1 ms)Running coprocessor post-close hooks at 1676310826616 (+13 ms)Closed at 1676310826617 (+1 ms) 2023-02-13 17:53:46,617 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(1837): Closed restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045. 2023-02-13 17:53:46,617 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:35949-0x1004ed10fed000a, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:46,617 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:46,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed restartTableThree,,1676310822711.f1791fa38e7b82b917ae39ff48365a4d. 2023-02-13 17:53:46,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(1557): Region close journal for f3a07b2ad33c42a90eb09beaac5e4045: Waiting for close lock at 1676310826601Running coprocessor pre-close hooks at 1676310826601Disabling compacts and flushes for region at 1676310826601Disabling writes for close at 1676310826602 (+1 ms)Writing region close event to WAL at 1676310826604 (+2 ms)Running coprocessor post-close hooks at 1676310826617 (+13 ms)Closed at 1676310826617 2023-02-13 17:53:46,617 ERROR [Listener at localhost/43465-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7d189f42 rejected from java.util.concurrent.ThreadPoolExecutor@79270439[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 10] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:46,617 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:46,618 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:35949-0] handler.CloseRegionHandler(117): Closed restartTableOne,,1676310821452.f3a07b2ad33c42a90eb09beaac5e4045. 2023-02-13 17:53:46,618 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 673bbd347d5066108d814a6b9fc9e57c, disabling compactions & flushes 2023-02-13 17:53:46,619 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676310820635.673bbd347d5066108d814a6b9fc9e57c. 2023-02-13 17:53:46,619 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(1603): Closing dc2f11e16ffd6ee6c6b4b51c20c090b8, disabling compactions & flushes 2023-02-13 17:53:46,619 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676310820635.673bbd347d5066108d814a6b9fc9e57c. 2023-02-13 17:53:46,619 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8. 2023-02-13 17:53:46,619 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676310820635.673bbd347d5066108d814a6b9fc9e57c. after waiting 0 ms 2023-02-13 17:53:46,620 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8. 2023-02-13 17:53:46,620 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676310820635.673bbd347d5066108d814a6b9fc9e57c. 2023-02-13 17:53:46,620 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8. after waiting 0 ms 2023-02-13 17:53:46,620 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 673bbd347d5066108d814a6b9fc9e57c 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-13 17:53:46,620 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8. 2023-02-13 17:53:46,624 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,34325,1676310818539] 2023-02-13 17:53:46,624 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,34325,1676310818539; numProcessing=1 2023-02-13 17:53:46,625 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,34325,1676310818539 already deleted, retry=false 2023-02-13 17:53:46,625 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,34325,1676310818539 expired; onlineServers=2 2023-02-13 17:53:46,626 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:35949-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/default/restartTableTwo/dc2f11e16ffd6ee6c6b4b51c20c090b8/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-13 17:53:46,627 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8. 2023-02-13 17:53:46,627 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:35949-0] regionserver.HRegion(1557): Region close journal for dc2f11e16ffd6ee6c6b4b51c20c090b8: Waiting for close lock at 1676310826619Running coprocessor pre-close hooks at 1676310826619Disabling compacts and flushes for region at 1676310826619Disabling writes for close at 1676310826620 (+1 ms)Writing region close event to WAL at 1676310826620Running coprocessor post-close hooks at 1676310826627 (+7 ms)Closed at 1676310826627 2023-02-13 17:53:46,627 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:35949-0] handler.CloseRegionHandler(117): Closed restartTableTwo,,1676310822093.dc2f11e16ffd6ee6c6b4b51c20c090b8. 2023-02-13 17:53:46,645 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.67 KB at sequenceid=28 (bloomFilter=false), to=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/meta/1588230740/.tmp/info/9661617356bb4b4fa4181b1a614e408a 2023-02-13 17:53:46,646 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/namespace/673bbd347d5066108d814a6b9fc9e57c/.tmp/info/56d97736c79e40f39dda0443b4055d1c 2023-02-13 17:53:46,652 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 17:53:46,652 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 17:53:46,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/namespace/673bbd347d5066108d814a6b9fc9e57c/.tmp/info/56d97736c79e40f39dda0443b4055d1c as hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/namespace/673bbd347d5066108d814a6b9fc9e57c/info/56d97736c79e40f39dda0443b4055d1c 2023-02-13 17:53:46,664 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/namespace/673bbd347d5066108d814a6b9fc9e57c/info/56d97736c79e40f39dda0443b4055d1c, entries=2, sequenceid=6, filesize=4.8 K 2023-02-13 17:53:46,667 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 673bbd347d5066108d814a6b9fc9e57c in 47ms, sequenceid=6, compaction requested=false 2023-02-13 17:53:46,678 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/namespace/673bbd347d5066108d814a6b9fc9e57c/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-13 17:53:46,680 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676310820635.673bbd347d5066108d814a6b9fc9e57c. 2023-02-13 17:53:46,680 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 673bbd347d5066108d814a6b9fc9e57c: Waiting for close lock at 1676310826618Running coprocessor pre-close hooks at 1676310826618Disabling compacts and flushes for region at 1676310826618Disabling writes for close at 1676310826620 (+2 ms)Obtaining lock to block concurrent updates at 1676310826620Preparing flush snapshotting stores in 673bbd347d5066108d814a6b9fc9e57c at 1676310826620Finished memstore snapshotting hbase:namespace,,1676310820635.673bbd347d5066108d814a6b9fc9e57c., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1676310826620Flushing stores of hbase:namespace,,1676310820635.673bbd347d5066108d814a6b9fc9e57c. at 1676310826621 (+1 ms)Flushing 673bbd347d5066108d814a6b9fc9e57c/info: creating writer at 1676310826621Flushing 673bbd347d5066108d814a6b9fc9e57c/info: appending metadata at 1676310826626 (+5 ms)Flushing 673bbd347d5066108d814a6b9fc9e57c/info: closing flushed file at 1676310826626Flushing 673bbd347d5066108d814a6b9fc9e57c/info: reopening flushed file at 1676310826657 (+31 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 673bbd347d5066108d814a6b9fc9e57c in 47ms, sequenceid=6, compaction requested=false at 1676310826667 (+10 ms)Writing region close event to WAL at 1676310826670 (+3 ms)Running coprocessor post-close hooks at 1676310826679 (+9 ms)Closed at 1676310826680 (+1 ms) 2023-02-13 17:53:46,680 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676310820635.673bbd347d5066108d814a6b9fc9e57c. 2023-02-13 17:53:46,795 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:34325-0x1004ed10fed0003, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:46,795 INFO [RS:2;jenkins-hbase4:34325] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,34325,1676310818539; zookeeper connection closed. 2023-02-13 17:53:46,795 ERROR [Listener at localhost/43465-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@34080093 rejected from java.util.concurrent.ThreadPoolExecutor@79270439[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 10] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:46,796 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:34325-0x1004ed10fed0003, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:46,796 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@33761f4] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@33761f4 2023-02-13 17:53:46,796 ERROR [Listener at localhost/43465-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4c14a090 rejected from java.util.concurrent.ThreadPoolExecutor@79270439[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 10] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:46,803 INFO [RS:3;jenkins-hbase4:35949] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,35949,1676310825652; all regions closed. 2023-02-13 17:53:46,804 DEBUG [RS:1;jenkins-hbase4:39175] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-02-13 17:53:46,810 DEBUG [RS:3;jenkins-hbase4:35949] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/oldWALs 2023-02-13 17:53:46,810 INFO [RS:3;jenkins-hbase4:35949] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C35949%2C1676310825652:(num 1676310826284) 2023-02-13 17:53:46,810 DEBUG [RS:3;jenkins-hbase4:35949] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:46,810 INFO [RS:3;jenkins-hbase4:35949] regionserver.LeaseManager(133): Closed leases 2023-02-13 17:53:46,811 INFO [RS:3;jenkins-hbase4:35949] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:35949 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 17:53:46,811 INFO [RS:3;jenkins-hbase4:35949] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 17:53:46,811 INFO [regionserver/jenkins-hbase4:35949.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 17:53:46,811 INFO [RS:3;jenkins-hbase4:35949] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 17:53:46,811 INFO [RS:3;jenkins-hbase4:35949] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 17:53:46,811 INFO [RS:3;jenkins-hbase4:35949] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:35949 2023-02-13 17:53:46,817 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:46,817 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,35949,1676310825652 2023-02-13 17:53:46,817 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:35949-0x1004ed10fed000a, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,35949,1676310825652 2023-02-13 17:53:46,817 ERROR [Listener at localhost/43465-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@33fff560 rejected from java.util.concurrent.ThreadPoolExecutor@37c077c7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:46,819 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,35949,1676310825652] 2023-02-13 17:53:46,819 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,35949,1676310825652; numProcessing=2 2023-02-13 17:53:46,918 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-13 17:53:46,918 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-13 17:53:46,919 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:35949-0x1004ed10fed000a, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:46,919 INFO [RS:3;jenkins-hbase4:35949] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,35949,1676310825652; zookeeper connection closed. 2023-02-13 17:53:46,919 ERROR [Listener at localhost/43465-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@f628810 rejected from java.util.concurrent.ThreadPoolExecutor@37c077c7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:46,919 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@c63d038] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@c63d038 2023-02-13 17:53:46,920 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:35949-0x1004ed10fed000a, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:46,920 ERROR [Listener at localhost/43465-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@77b0019e rejected from java.util.concurrent.ThreadPoolExecutor@37c077c7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:46,922 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,35949,1676310825652 already deleted, retry=false 2023-02-13 17:53:46,922 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,35949,1676310825652 expired; onlineServers=1 2023-02-13 17:53:47,004 DEBUG [RS:1;jenkins-hbase4:39175] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-02-13 17:53:47,077 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=380 B at sequenceid=28 (bloomFilter=false), to=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/meta/1588230740/.tmp/table/4f99398b08bf4c888a4332220f7ff347 2023-02-13 17:53:47,085 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/meta/1588230740/.tmp/info/9661617356bb4b4fa4181b1a614e408a as hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/meta/1588230740/info/9661617356bb4b4fa4181b1a614e408a 2023-02-13 17:53:47,091 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/meta/1588230740/info/9661617356bb4b4fa4181b1a614e408a, entries=48, sequenceid=28, filesize=10.4 K 2023-02-13 17:53:47,092 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/meta/1588230740/.tmp/table/4f99398b08bf4c888a4332220f7ff347 as hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/meta/1588230740/table/4f99398b08bf4c888a4332220f7ff347 2023-02-13 17:53:47,099 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/meta/1588230740/table/4f99398b08bf4c888a4332220f7ff347, entries=8, sequenceid=28, filesize=5.0 K 2023-02-13 17:53:47,100 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~7.04 KB/7210, heapSize ~12.22 KB/12512, currentSize=0 B/0 for 1588230740 in 494ms, sequenceid=28, compaction requested=false 2023-02-13 17:53:47,107 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/data/hbase/meta/1588230740/recovered.edits/31.seqid, newMaxSeqId=31, maxSeqId=1 2023-02-13 17:53:47,107 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 17:53:47,108 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 17:53:47,108 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676310826605Running coprocessor pre-close hooks at 1676310826605Disabling compacts and flushes for region at 1676310826605Disabling writes for close at 1676310826606 (+1 ms)Obtaining lock to block concurrent updates at 1676310826606Preparing flush snapshotting stores in 1588230740 at 1676310826606Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=7210, getHeapSize=12752, getOffHeapSize=0, getCellsCount=64 at 1676310826608 (+2 ms)Flushing stores of hbase:meta,,1.1588230740 at 1676310826609 (+1 ms)Flushing 1588230740/info: creating writer at 1676310826610 (+1 ms)Flushing 1588230740/info: appending metadata at 1676310826621 (+11 ms)Flushing 1588230740/info: closing flushed file at 1676310826621Flushing 1588230740/table: creating writer at 1676310826654 (+33 ms)Flushing 1588230740/table: appending metadata at 1676310826659 (+5 ms)Flushing 1588230740/table: closing flushed file at 1676310826659Flushing 1588230740/info: reopening flushed file at 1676310827085 (+426 ms)Flushing 1588230740/table: reopening flushed file at 1676310827093 (+8 ms)Finished flush of dataSize ~7.04 KB/7210, heapSize ~12.22 KB/12512, currentSize=0 B/0 for 1588230740 in 494ms, sequenceid=28, compaction requested=false at 1676310827100 (+7 ms)Writing region close event to WAL at 1676310827102 (+2 ms)Running coprocessor post-close hooks at 1676310827107 (+5 ms)Closed at 1676310827108 (+1 ms) 2023-02-13 17:53:47,108 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-13 17:53:47,205 INFO [RS:1;jenkins-hbase4:39175] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,39175,1676310818498; all regions closed. 2023-02-13 17:53:47,212 DEBUG [RS:1;jenkins-hbase4:39175] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/oldWALs 2023-02-13 17:53:47,212 INFO [RS:1;jenkins-hbase4:39175] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C39175%2C1676310818498.meta:.meta(num 1676310820369) 2023-02-13 17:53:47,218 DEBUG [RS:1;jenkins-hbase4:39175] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/oldWALs 2023-02-13 17:53:47,218 INFO [RS:1;jenkins-hbase4:39175] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C39175%2C1676310818498:(num 1676310820087) 2023-02-13 17:53:47,218 DEBUG [RS:1;jenkins-hbase4:39175] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:47,218 INFO [RS:1;jenkins-hbase4:39175] regionserver.LeaseManager(133): Closed leases 2023-02-13 17:53:47,218 INFO [RS:1;jenkins-hbase4:39175] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-13 17:53:47,218 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 17:53:47,219 INFO [RS:1;jenkins-hbase4:39175] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39175 2023-02-13 17:53:47,221 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39175,1676310818498 2023-02-13 17:53:47,221 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:47,221 ERROR [Listener at localhost/43465-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@f496971 rejected from java.util.concurrent.ThreadPoolExecutor@b76ec40[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 12] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1374) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:47,222 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,39175,1676310818498] 2023-02-13 17:53:47,222 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,39175,1676310818498; numProcessing=3 2023-02-13 17:53:47,223 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,39175,1676310818498 already deleted, retry=false 2023-02-13 17:53:47,223 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,39175,1676310818498 expired; onlineServers=0 2023-02-13 17:53:47,224 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,37439,1676310825610' ***** 2023-02-13 17:53:47,224 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-13 17:53:47,224 DEBUG [M:0;jenkins-hbase4:37439] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@27c232d7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 17:53:47,224 INFO [M:0;jenkins-hbase4:37439] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,37439,1676310825610 2023-02-13 17:53:47,224 INFO [M:0;jenkins-hbase4:37439] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,37439,1676310825610; all regions closed. 2023-02-13 17:53:47,224 DEBUG [M:0;jenkins-hbase4:37439] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:47,224 DEBUG [M:0;jenkins-hbase4:37439] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-13 17:53:47,224 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-13 17:53:47,224 DEBUG [M:0;jenkins-hbase4:37439] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-13 17:53:47,224 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676310825927] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676310825927,5,FailOnTimeoutGroup] 2023-02-13 17:53:47,224 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676310825928] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676310825928,5,FailOnTimeoutGroup] 2023-02-13 17:53:47,225 INFO [M:0;jenkins-hbase4:37439] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-13 17:53:47,226 INFO [M:0;jenkins-hbase4:37439] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-13 17:53:47,226 INFO [M:0;jenkins-hbase4:37439] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-13 17:53:47,226 DEBUG [M:0;jenkins-hbase4:37439] master.HMaster(1502): Stopping service threads 2023-02-13 17:53:47,226 INFO [M:0;jenkins-hbase4:37439] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-13 17:53:47,226 ERROR [M:0;jenkins-hbase4:37439] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-13 17:53:47,226 INFO [M:0;jenkins-hbase4:37439] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-13 17:53:47,226 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-13 17:53:47,227 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 17:53:47,227 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:53:47,227 DEBUG [M:0;jenkins-hbase4:37439] zookeeper.ZKUtil(398): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-13 17:53:47,227 WARN [M:0;jenkins-hbase4:37439] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-13 17:53:47,227 INFO [M:0;jenkins-hbase4:37439] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-13 17:53:47,228 INFO [M:0;jenkins-hbase4:37439] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-13 17:53:47,228 DEBUG [M:0;jenkins-hbase4:37439] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 17:53:47,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 17:53:47,228 INFO [M:0;jenkins-hbase4:37439] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:53:47,228 DEBUG [M:0;jenkins-hbase4:37439] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:53:47,228 DEBUG [M:0;jenkins-hbase4:37439] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 17:53:47,229 DEBUG [M:0;jenkins-hbase4:37439] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:53:47,229 INFO [M:0;jenkins-hbase4:37439] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=14.68 KB heapSize=18.26 KB 2023-02-13 17:53:47,246 INFO [M:0;jenkins-hbase4:37439] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=14.68 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/9967e7ebc09c48a8a0f3e3ea378f10ce 2023-02-13 17:53:47,252 DEBUG [M:0;jenkins-hbase4:37439] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/9967e7ebc09c48a8a0f3e3ea378f10ce as hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/9967e7ebc09c48a8a0f3e3ea378f10ce 2023-02-13 17:53:47,258 INFO [M:0;jenkins-hbase4:37439] regionserver.HStore(1080): Added hdfs://localhost:42429/user/jenkins/test-data/37ae2da4-4c5b-40ab-f0d8-0bd1015a8de2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/9967e7ebc09c48a8a0f3e3ea378f10ce, entries=5, sequenceid=182, filesize=5.3 K 2023-02-13 17:53:47,259 INFO [M:0;jenkins-hbase4:37439] regionserver.HRegion(2947): Finished flush of dataSize ~14.68 KB/15031, heapSize ~18.24 KB/18680, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 30ms, sequenceid=182, compaction requested=false 2023-02-13 17:53:47,261 INFO [M:0;jenkins-hbase4:37439] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:53:47,261 DEBUG [M:0;jenkins-hbase4:37439] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676310827228Disabling compacts and flushes for region at 1676310827228Disabling writes for close at 1676310827229 (+1 ms)Obtaining lock to block concurrent updates at 1676310827229Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676310827229Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=15031, getHeapSize=18680, getOffHeapSize=0, getCellsCount=45 at 1676310827229Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676310827230 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676310827230Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676310827233 (+3 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676310827234 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676310827253 (+19 ms)Finished flush of dataSize ~14.68 KB/15031, heapSize ~18.24 KB/18680, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 30ms, sequenceid=182, compaction requested=false at 1676310827259 (+6 ms)Writing region close event to WAL at 1676310827261 (+2 ms)Closed at 1676310827261 2023-02-13 17:53:47,265 INFO [M:0;jenkins-hbase4:37439] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-13 17:53:47,265 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 17:53:47,265 INFO [M:0;jenkins-hbase4:37439] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:37439 2023-02-13 17:53:47,269 DEBUG [M:0;jenkins-hbase4:37439] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,37439,1676310825610 already deleted, retry=false 2023-02-13 17:53:47,323 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:47,323 INFO [RS:1;jenkins-hbase4:39175] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,39175,1676310818498; zookeeper connection closed. 2023-02-13 17:53:47,323 ERROR [Listener at localhost/43465-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3b36dab3 rejected from java.util.concurrent.ThreadPoolExecutor@b76ec40[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 12] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:47,323 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): regionserver:39175-0x1004ed10fed0002, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:47,323 ERROR [Listener at localhost/43465-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@43d8c907 rejected from java.util.concurrent.ThreadPoolExecutor@b76ec40[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 12] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:47,323 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@447b3584] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@447b3584 2023-02-13 17:53:47,324 INFO [Listener at localhost/43465] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 4 regionserver(s) complete 2023-02-13 17:53:47,423 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:47,423 INFO [M:0;jenkins-hbase4:37439] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,37439,1676310825610; zookeeper connection closed. 2023-02-13 17:53:47,423 ERROR [Listener at localhost/43465-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@62cf6e4e rejected from java.util.concurrent.ThreadPoolExecutor@66f0e6cf[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 16] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:47,424 DEBUG [Listener at localhost/43465-EventThread] zookeeper.ZKWatcher(600): master:37439-0x1004ed10fed0009, quorum=127.0.0.1:52890, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:47,424 ERROR [Listener at localhost/43465-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@b3f70c1 rejected from java.util.concurrent.ThreadPoolExecutor@66f0e6cf[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 16] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:47,425 WARN [Listener at localhost/43465] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 17:53:47,429 INFO [Listener at localhost/43465] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 17:53:47,533 WARN [BP-1718455074-172.31.14.131-1676310813473 heartbeating to localhost/127.0.0.1:42429] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 17:53:47,533 WARN [BP-1718455074-172.31.14.131-1676310813473 heartbeating to localhost/127.0.0.1:42429] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1718455074-172.31.14.131-1676310813473 (Datanode Uuid c7394c73-8463-404a-9d61-e450c05adbdc) service to localhost/127.0.0.1:42429 2023-02-13 17:53:47,535 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/cluster_76469730-fa10-8e70-bc1e-b5e0ec8a135a/dfs/data/data5/current/BP-1718455074-172.31.14.131-1676310813473] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 17:53:47,535 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/cluster_76469730-fa10-8e70-bc1e-b5e0ec8a135a/dfs/data/data6/current/BP-1718455074-172.31.14.131-1676310813473] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 17:53:47,537 WARN [Listener at localhost/43465] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 17:53:47,540 INFO [Listener at localhost/43465] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 17:53:47,644 WARN [BP-1718455074-172.31.14.131-1676310813473 heartbeating to localhost/127.0.0.1:42429] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 17:53:47,644 WARN [BP-1718455074-172.31.14.131-1676310813473 heartbeating to localhost/127.0.0.1:42429] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1718455074-172.31.14.131-1676310813473 (Datanode Uuid ce050643-8f71-4f57-ad61-72ab186e54bd) service to localhost/127.0.0.1:42429 2023-02-13 17:53:47,644 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/cluster_76469730-fa10-8e70-bc1e-b5e0ec8a135a/dfs/data/data3/current/BP-1718455074-172.31.14.131-1676310813473] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 17:53:47,645 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/cluster_76469730-fa10-8e70-bc1e-b5e0ec8a135a/dfs/data/data4/current/BP-1718455074-172.31.14.131-1676310813473] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 17:53:47,646 WARN [Listener at localhost/43465] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 17:53:47,648 INFO [Listener at localhost/43465] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 17:53:47,750 WARN [BP-1718455074-172.31.14.131-1676310813473 heartbeating to localhost/127.0.0.1:42429] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 17:53:47,750 WARN [BP-1718455074-172.31.14.131-1676310813473 heartbeating to localhost/127.0.0.1:42429] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1718455074-172.31.14.131-1676310813473 (Datanode Uuid ac9475ad-04ff-47e5-82bc-722ccbf8b5e7) service to localhost/127.0.0.1:42429 2023-02-13 17:53:47,751 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/cluster_76469730-fa10-8e70-bc1e-b5e0ec8a135a/dfs/data/data1/current/BP-1718455074-172.31.14.131-1676310813473] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 17:53:47,752 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/cluster_76469730-fa10-8e70-bc1e-b5e0ec8a135a/dfs/data/data2/current/BP-1718455074-172.31.14.131-1676310813473] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 17:53:47,786 INFO [Listener at localhost/43465] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 17:53:47,904 INFO [Listener at localhost/43465] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-13 17:53:47,958 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-02-13 17:53:47,973 INFO [Listener at localhost/43465] hbase.ResourceChecker(175): after: master.TestRetainAssignmentOnRestartSplitWithoutZk#testRetainAssignmentOnSingleRSRestart Thread=99 (was 7) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (211326415) connection to localhost/127.0.0.1:42429 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: region-location-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-1 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-6-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost/43465 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (211326415) connection to localhost/127.0.0.1:42429 from jenkins.hfs.0 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: master/jenkins-hbase4:0:becomeActiveMaster-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-7-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-4-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:35949.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: ForkJoinPool-2-worker-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: nioEventLoopGroup-7-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (211326415) connection to localhost/127.0.0.1:42429 from jenkins.hfs.4 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: ForkJoinPool-2-worker-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: nioEventLoopGroup-6-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: nioEventLoopGroup-7-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: nioEventLoopGroup-4-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReplicationExecutor-0 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:703) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-3-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3693) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.3@localhost:42429 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-6-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: SnapshotHandlerChoreCleaner sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: nioEventLoopGroup-3-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (211326415) connection to localhost/127.0.0.1:42429 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (211326415) connection to localhost/127.0.0.1:42429 from jenkins.hfs.3 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-4-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReplicationExecutor-0 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:703) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcClient-timer-pool-0 java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:600) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:496) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.0@localhost:42429 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.4@localhost:42429 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Monitor thread for TaskMonitor java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.monitoring.TaskMonitor$MonitorRunnable.run(TaskMonitor.java:300) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HBase-Metrics2-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (211326415) connection to localhost/127.0.0.1:42429 from jenkins.hfs.1 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: SnapshotHandlerChoreCleaner sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: nioEventLoopGroup-2-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Idle-Rpc-Conn-Sweeper-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:42429 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: master/jenkins-hbase4:0:becomeActiveMaster-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: SessionTracker java.lang.Thread.sleep(Native Method) org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:151) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-3-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: org.apache.hadoop.hdfs.PeerCache@64eb93cf java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:253) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (211326415) connection to localhost/127.0.0.1:42429 from jenkins.hfs.2 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: LeaseRenewer:jenkins.hfs.1@localhost:42429 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins@localhost:42429 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=504 (was 260) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=379 (was 389), ProcessCount=168 (was 168), AvailableMemoryMB=1400 (was 2840) 2023-02-13 17:53:47,983 INFO [Listener at localhost/43465] hbase.ResourceChecker(147): before: master.TestRetainAssignmentOnRestartSplitWithoutZk#testRetainAssignmentOnClusterRestart Thread=99, OpenFileDescriptor=504, MaxFileDescriptor=60000, SystemLoadAverage=379, ProcessCount=168, AvailableMemoryMB=1399 2023-02-13 17:53:47,995 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.log.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/hadoop.log.dir so I do NOT create it in target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14 2023-02-13 17:53:47,995 WARN [Listener at localhost/43465] hbase.HBaseTestingUtility(449): hadoop.log.dir property value differs in configuration and system: Configuration=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/../logs while System=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/hadoop.log.dir Erasing configuration value by system value. 2023-02-13 17:53:47,995 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.tmp.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/hadoop.tmp.dir so I do NOT create it in target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14 2023-02-13 17:53:47,996 WARN [Listener at localhost/43465] hbase.HBaseTestingUtility(449): hadoop.tmp.dir property value differs in configuration and system: Configuration=/tmp/hadoop-jenkins while System=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89b52c47-b07f-7f5f-0839-f3777a47aa06/hadoop.tmp.dir Erasing configuration value by system value. 2023-02-13 17:53:47,996 DEBUG [Listener at localhost/43465] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14 2023-02-13 17:53:47,996 INFO [Listener at localhost/43465] master.AbstractTestRestartCluster(45): WAL splitting coordinated by zk false 2023-02-13 17:53:47,996 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-13 17:53:47,996 INFO [Listener at localhost/43465] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/cluster_5150fa22-ca9c-c15d-e00c-0d4af63a07cb, deleteOnExit=true 2023-02-13 17:53:47,996 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-13 17:53:47,996 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/test.cache.data in system properties and HBase conf 2023-02-13 17:53:47,996 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/hadoop.tmp.dir in system properties and HBase conf 2023-02-13 17:53:47,996 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/hadoop.log.dir in system properties and HBase conf 2023-02-13 17:53:47,997 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-13 17:53:47,997 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-13 17:53:47,997 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-13 17:53:47,997 DEBUG [Listener at localhost/43465] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-13 17:53:47,997 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-13 17:53:47,997 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-13 17:53:47,997 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-13 17:53:47,997 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-13 17:53:47,998 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-13 17:53:47,998 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-13 17:53:47,998 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-13 17:53:47,998 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-13 17:53:47,998 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-13 17:53:47,998 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/nfs.dump.dir in system properties and HBase conf 2023-02-13 17:53:47,998 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/java.io.tmpdir in system properties and HBase conf 2023-02-13 17:53:47,998 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-13 17:53:47,998 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-13 17:53:47,998 INFO [Listener at localhost/43465] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-13 17:53:48,003 WARN [Listener at localhost/43465] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-13 17:53:48,003 WARN [Listener at localhost/43465] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-13 17:53:48,051 WARN [Listener at localhost/43465] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 17:53:48,053 INFO [Listener at localhost/43465] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 17:53:48,057 INFO [Listener at localhost/43465] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/java.io.tmpdir/Jetty_localhost_43977_hdfs____.mbb7mk/webapp 2023-02-13 17:53:48,150 INFO [Listener at localhost/43465] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43977 2023-02-13 17:53:48,154 WARN [Listener at localhost/43465] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-13 17:53:48,154 WARN [Listener at localhost/43465] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-13 17:53:48,197 WARN [Listener at localhost/36959] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 17:53:48,209 WARN [Listener at localhost/36959] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 17:53:48,211 WARN [Listener at localhost/36959] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 17:53:48,212 INFO [Listener at localhost/36959] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 17:53:48,217 INFO [Listener at localhost/36959] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/java.io.tmpdir/Jetty_localhost_33191_datanode____.ipk4q8/webapp 2023-02-13 17:53:48,308 INFO [Listener at localhost/36959] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33191 2023-02-13 17:53:48,315 WARN [Listener at localhost/45345] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 17:53:48,328 WARN [Listener at localhost/45345] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 17:53:48,330 WARN [Listener at localhost/45345] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 17:53:48,331 INFO [Listener at localhost/45345] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 17:53:48,334 INFO [Listener at localhost/45345] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/java.io.tmpdir/Jetty_localhost_33707_datanode____yyf6dh/webapp 2023-02-13 17:53:48,438 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xfbdcfa8e5f74e666: Processing first storage report for DS-c79d1410-43a1-4b12-a2c2-dff1472abceb from datanode fa18d43c-49c0-4428-a5dc-0045e2c3f0c7 2023-02-13 17:53:48,438 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xfbdcfa8e5f74e666: from storage DS-c79d1410-43a1-4b12-a2c2-dff1472abceb node DatanodeRegistration(127.0.0.1:37727, datanodeUuid=fa18d43c-49c0-4428-a5dc-0045e2c3f0c7, infoPort=42513, infoSecurePort=0, ipcPort=45345, storageInfo=lv=-57;cid=testClusterID;nsid=767618815;c=1676310828006), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 17:53:48,438 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xfbdcfa8e5f74e666: Processing first storage report for DS-5a96b79c-bee8-4ee4-95ce-38e1036c3cca from datanode fa18d43c-49c0-4428-a5dc-0045e2c3f0c7 2023-02-13 17:53:48,438 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xfbdcfa8e5f74e666: from storage DS-5a96b79c-bee8-4ee4-95ce-38e1036c3cca node DatanodeRegistration(127.0.0.1:37727, datanodeUuid=fa18d43c-49c0-4428-a5dc-0045e2c3f0c7, infoPort=42513, infoSecurePort=0, ipcPort=45345, storageInfo=lv=-57;cid=testClusterID;nsid=767618815;c=1676310828006), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 17:53:48,451 INFO [Listener at localhost/45345] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33707 2023-02-13 17:53:48,460 WARN [Listener at localhost/33131] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 17:53:48,479 WARN [Listener at localhost/33131] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 17:53:48,481 WARN [Listener at localhost/33131] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 17:53:48,482 INFO [Listener at localhost/33131] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 17:53:48,486 INFO [Listener at localhost/33131] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/java.io.tmpdir/Jetty_localhost_35345_datanode____.yd8afb/webapp 2023-02-13 17:53:48,605 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xe02c7f9fa26e99ad: Processing first storage report for DS-cf1e0e36-9598-4b66-ab37-608d409fcf08 from datanode 8d40f6dd-bd2c-4b79-b3dd-37281edb3cc3 2023-02-13 17:53:48,605 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xe02c7f9fa26e99ad: from storage DS-cf1e0e36-9598-4b66-ab37-608d409fcf08 node DatanodeRegistration(127.0.0.1:39523, datanodeUuid=8d40f6dd-bd2c-4b79-b3dd-37281edb3cc3, infoPort=42113, infoSecurePort=0, ipcPort=33131, storageInfo=lv=-57;cid=testClusterID;nsid=767618815;c=1676310828006), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 17:53:48,605 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xe02c7f9fa26e99ad: Processing first storage report for DS-28042078-d218-4ed6-b78e-cd1ad3e28bf7 from datanode 8d40f6dd-bd2c-4b79-b3dd-37281edb3cc3 2023-02-13 17:53:48,605 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xe02c7f9fa26e99ad: from storage DS-28042078-d218-4ed6-b78e-cd1ad3e28bf7 node DatanodeRegistration(127.0.0.1:39523, datanodeUuid=8d40f6dd-bd2c-4b79-b3dd-37281edb3cc3, infoPort=42113, infoSecurePort=0, ipcPort=33131, storageInfo=lv=-57;cid=testClusterID;nsid=767618815;c=1676310828006), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 17:53:48,625 INFO [Listener at localhost/33131] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35345 2023-02-13 17:53:48,636 WARN [Listener at localhost/40181] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 17:53:48,753 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x716dc6f4f5f2814b: Processing first storage report for DS-c817cfbe-f58e-4f4d-94e0-14281a32ca4c from datanode 5b44eaac-9553-466f-8994-1edc7d6b673f 2023-02-13 17:53:48,753 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x716dc6f4f5f2814b: from storage DS-c817cfbe-f58e-4f4d-94e0-14281a32ca4c node DatanodeRegistration(127.0.0.1:37061, datanodeUuid=5b44eaac-9553-466f-8994-1edc7d6b673f, infoPort=42967, infoSecurePort=0, ipcPort=40181, storageInfo=lv=-57;cid=testClusterID;nsid=767618815;c=1676310828006), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 17:53:48,753 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x716dc6f4f5f2814b: Processing first storage report for DS-dbfbb1ac-54cb-487d-a0fc-2eb9fde1b490 from datanode 5b44eaac-9553-466f-8994-1edc7d6b673f 2023-02-13 17:53:48,753 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x716dc6f4f5f2814b: from storage DS-dbfbb1ac-54cb-487d-a0fc-2eb9fde1b490 node DatanodeRegistration(127.0.0.1:37061, datanodeUuid=5b44eaac-9553-466f-8994-1edc7d6b673f, infoPort=42967, infoSecurePort=0, ipcPort=40181, storageInfo=lv=-57;cid=testClusterID;nsid=767618815;c=1676310828006), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 17:53:48,852 DEBUG [Listener at localhost/40181] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14 2023-02-13 17:53:48,856 INFO [Listener at localhost/40181] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/cluster_5150fa22-ca9c-c15d-e00c-0d4af63a07cb/zookeeper_0, clientPort=53630, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/cluster_5150fa22-ca9c-c15d-e00c-0d4af63a07cb/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/cluster_5150fa22-ca9c-c15d-e00c-0d4af63a07cb/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-13 17:53:48,858 INFO [Listener at localhost/40181] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=53630 2023-02-13 17:53:48,858 INFO [Listener at localhost/40181] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:48,859 INFO [Listener at localhost/40181] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:48,879 INFO [Listener at localhost/40181] util.FSUtils(479): Created version file at hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e with version=8 2023-02-13 17:53:48,879 INFO [Listener at localhost/40181] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/hbase-staging 2023-02-13 17:53:48,881 INFO [Listener at localhost/40181] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 17:53:48,882 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:48,882 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:48,882 INFO [Listener at localhost/40181] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 17:53:48,882 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:48,882 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 17:53:48,882 INFO [Listener at localhost/40181] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 17:53:48,883 INFO [Listener at localhost/40181] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46373 2023-02-13 17:53:48,884 INFO [Listener at localhost/40181] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:48,885 INFO [Listener at localhost/40181] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:48,886 INFO [Listener at localhost/40181] zookeeper.RecoverableZooKeeper(93): Process identifier=master:46373 connecting to ZooKeeper ensemble=127.0.0.1:53630 2023-02-13 17:53:48,894 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:463730x0, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 17:53:48,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:46373-0x1004ed140fc0000 connected 2023-02-13 17:53:48,908 DEBUG [Listener at localhost/40181] zookeeper.ZKUtil(164): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 17:53:48,908 DEBUG [Listener at localhost/40181] zookeeper.ZKUtil(164): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 17:53:48,909 DEBUG [Listener at localhost/40181] zookeeper.ZKUtil(164): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 17:53:48,909 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46373 2023-02-13 17:53:48,909 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46373 2023-02-13 17:53:48,910 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46373 2023-02-13 17:53:48,910 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46373 2023-02-13 17:53:48,910 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46373 2023-02-13 17:53:48,910 INFO [Listener at localhost/40181] master.HMaster(439): hbase.rootdir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e, hbase.cluster.distributed=false 2023-02-13 17:53:48,923 INFO [Listener at localhost/40181] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 17:53:48,923 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:48,924 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:48,924 INFO [Listener at localhost/40181] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 17:53:48,924 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:48,924 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 17:53:48,924 INFO [Listener at localhost/40181] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 17:53:48,925 INFO [Listener at localhost/40181] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39461 2023-02-13 17:53:48,925 INFO [Listener at localhost/40181] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 17:53:48,927 DEBUG [Listener at localhost/40181] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 17:53:48,927 INFO [Listener at localhost/40181] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:48,928 INFO [Listener at localhost/40181] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:48,929 INFO [Listener at localhost/40181] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39461 connecting to ZooKeeper ensemble=127.0.0.1:53630 2023-02-13 17:53:48,934 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:394610x0, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 17:53:48,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:39461-0x1004ed140fc0001 connected 2023-02-13 17:53:48,935 DEBUG [Listener at localhost/40181] zookeeper.ZKUtil(164): regionserver:39461-0x1004ed140fc0001, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 17:53:48,936 DEBUG [Listener at localhost/40181] zookeeper.ZKUtil(164): regionserver:39461-0x1004ed140fc0001, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 17:53:48,936 DEBUG [Listener at localhost/40181] zookeeper.ZKUtil(164): regionserver:39461-0x1004ed140fc0001, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 17:53:48,940 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39461 2023-02-13 17:53:48,940 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39461 2023-02-13 17:53:48,940 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39461 2023-02-13 17:53:48,940 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39461 2023-02-13 17:53:48,941 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39461 2023-02-13 17:53:48,952 INFO [Listener at localhost/40181] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 17:53:48,952 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:48,952 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:48,952 INFO [Listener at localhost/40181] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 17:53:48,952 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:48,952 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 17:53:48,952 INFO [Listener at localhost/40181] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 17:53:48,953 INFO [Listener at localhost/40181] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34875 2023-02-13 17:53:48,954 INFO [Listener at localhost/40181] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 17:53:48,955 DEBUG [Listener at localhost/40181] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 17:53:48,955 INFO [Listener at localhost/40181] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:48,956 INFO [Listener at localhost/40181] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:48,957 INFO [Listener at localhost/40181] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34875 connecting to ZooKeeper ensemble=127.0.0.1:53630 2023-02-13 17:53:48,962 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:348750x0, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 17:53:48,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:34875-0x1004ed140fc0002 connected 2023-02-13 17:53:48,964 DEBUG [Listener at localhost/40181] zookeeper.ZKUtil(164): regionserver:34875-0x1004ed140fc0002, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 17:53:48,964 DEBUG [Listener at localhost/40181] zookeeper.ZKUtil(164): regionserver:34875-0x1004ed140fc0002, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 17:53:48,965 DEBUG [Listener at localhost/40181] zookeeper.ZKUtil(164): regionserver:34875-0x1004ed140fc0002, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 17:53:48,968 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34875 2023-02-13 17:53:48,969 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34875 2023-02-13 17:53:48,969 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34875 2023-02-13 17:53:48,970 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34875 2023-02-13 17:53:48,970 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34875 2023-02-13 17:53:48,981 INFO [Listener at localhost/40181] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 17:53:48,981 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:48,981 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:48,981 INFO [Listener at localhost/40181] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 17:53:48,982 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:48,982 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 17:53:48,982 INFO [Listener at localhost/40181] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 17:53:48,983 INFO [Listener at localhost/40181] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42905 2023-02-13 17:53:48,983 INFO [Listener at localhost/40181] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 17:53:48,987 DEBUG [Listener at localhost/40181] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 17:53:48,987 INFO [Listener at localhost/40181] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:48,988 INFO [Listener at localhost/40181] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:48,989 INFO [Listener at localhost/40181] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:42905 connecting to ZooKeeper ensemble=127.0.0.1:53630 2023-02-13 17:53:48,992 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:429050x0, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 17:53:48,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:42905-0x1004ed140fc0003 connected 2023-02-13 17:53:48,993 DEBUG [Listener at localhost/40181] zookeeper.ZKUtil(164): regionserver:42905-0x1004ed140fc0003, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 17:53:48,994 DEBUG [Listener at localhost/40181] zookeeper.ZKUtil(164): regionserver:42905-0x1004ed140fc0003, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 17:53:48,994 DEBUG [Listener at localhost/40181] zookeeper.ZKUtil(164): regionserver:42905-0x1004ed140fc0003, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 17:53:48,998 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42905 2023-02-13 17:53:48,998 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42905 2023-02-13 17:53:49,000 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42905 2023-02-13 17:53:49,003 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42905 2023-02-13 17:53:49,004 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42905 2023-02-13 17:53:49,006 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,46373,1676310828881 2023-02-13 17:53:49,007 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 17:53:49,008 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,46373,1676310828881 2023-02-13 17:53:49,010 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:39461-0x1004ed140fc0001, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 17:53:49,010 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:42905-0x1004ed140fc0003, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 17:53:49,010 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:34875-0x1004ed140fc0002, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 17:53:49,010 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 17:53:49,011 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:53:49,012 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 17:53:49,015 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,46373,1676310828881 from backup master directory 2023-02-13 17:53:49,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 17:53:49,016 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,46373,1676310828881 2023-02-13 17:53:49,016 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 17:53:49,016 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 17:53:49,017 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,46373,1676310828881 2023-02-13 17:53:49,041 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/hbase.id with ID: 2a4478ba-ba99-441c-8812-a0c5f19f8b8a 2023-02-13 17:53:49,060 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:49,063 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:53:49,085 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x28d37515 to 127.0.0.1:53630 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:53:49,089 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d7bde8c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:53:49,089 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 17:53:49,089 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-13 17:53:49,090 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 17:53:49,091 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/data/master/store-tmp 2023-02-13 17:53:49,108 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:49,108 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 17:53:49,108 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:53:49,108 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:53:49,108 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 17:53:49,108 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:53:49,108 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:53:49,108 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676310829108Disabling compacts and flushes for region at 1676310829108Disabling writes for close at 1676310829108Writing region close event to WAL at 1676310829108Closed at 1676310829108 2023-02-13 17:53:49,109 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/WALs/jenkins-hbase4.apache.org,46373,1676310828881 2023-02-13 17:53:49,112 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46373%2C1676310828881, suffix=, logDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/WALs/jenkins-hbase4.apache.org,46373,1676310828881, archiveDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/oldWALs, maxLogs=10 2023-02-13 17:53:49,134 DEBUG [RS-EventLoopGroup-12-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39523,DS-cf1e0e36-9598-4b66-ab37-608d409fcf08,DISK] 2023-02-13 17:53:49,134 DEBUG [RS-EventLoopGroup-12-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37727,DS-c79d1410-43a1-4b12-a2c2-dff1472abceb,DISK] 2023-02-13 17:53:49,136 DEBUG [RS-EventLoopGroup-12-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37061,DS-c817cfbe-f58e-4f4d-94e0-14281a32ca4c,DISK] 2023-02-13 17:53:49,139 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/WALs/jenkins-hbase4.apache.org,46373,1676310828881/jenkins-hbase4.apache.org%2C46373%2C1676310828881.1676310829113 2023-02-13 17:53:49,139 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39523,DS-cf1e0e36-9598-4b66-ab37-608d409fcf08,DISK], DatanodeInfoWithStorage[127.0.0.1:37727,DS-c79d1410-43a1-4b12-a2c2-dff1472abceb,DISK], DatanodeInfoWithStorage[127.0.0.1:37061,DS-c817cfbe-f58e-4f4d-94e0-14281a32ca4c,DISK]] 2023-02-13 17:53:49,139 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-13 17:53:49,139 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:49,139 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 17:53:49,140 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 17:53:49,142 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-13 17:53:49,144 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-13 17:53:49,145 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-13 17:53:49,145 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:49,146 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-13 17:53:49,147 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-13 17:53:49,150 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 17:53:49,152 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:53:49,153 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64426383, jitterRate=-0.039972081780433655}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:53:49,153 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676310829140Initializing all the Stores at 1676310829141 (+1 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676310829141Cleaning up temporary data from old regions at 1676310829147 (+6 ms)Cleaning up detritus from prior splits at 1676310829148 (+1 ms)Region opened successfully at 1676310829153 (+5 ms) 2023-02-13 17:53:49,153 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-13 17:53:49,155 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-13 17:53:49,155 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-13 17:53:49,155 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-13 17:53:49,155 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-13 17:53:49,156 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-02-13 17:53:49,156 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-13 17:53:49,161 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-13 17:53:49,162 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-13 17:53:49,172 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-13 17:53:49,172 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 17:53:49,173 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-13 17:53:49,173 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-13 17:53:49,174 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-13 17:53:49,176 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:53:49,177 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-13 17:53:49,177 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-13 17:53:49,178 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-13 17:53:49,179 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:42905-0x1004ed140fc0003, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 17:53:49,179 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:39461-0x1004ed140fc0001, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 17:53:49,179 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 17:53:49,179 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:34875-0x1004ed140fc0002, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 17:53:49,179 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:53:49,180 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,46373,1676310828881, sessionid=0x1004ed140fc0000, setting cluster-up flag (Was=false) 2023-02-13 17:53:49,184 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:53:49,194 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-13 17:53:49,195 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,46373,1676310828881 2023-02-13 17:53:49,198 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:53:49,202 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-13 17:53:49,202 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,46373,1676310828881 2023-02-13 17:53:49,203 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/.hbase-snapshot/.tmp 2023-02-13 17:53:49,205 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-13 17:53:49,205 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 17:53:49,205 INFO [RS:0;jenkins-hbase4:39461] regionserver.HRegionServer(952): ClusterId : 2a4478ba-ba99-441c-8812-a0c5f19f8b8a 2023-02-13 17:53:49,205 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 17:53:49,205 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 17:53:49,205 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 17:53:49,205 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-13 17:53:49,205 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,205 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 17:53:49,206 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,207 DEBUG [RS:0;jenkins-hbase4:39461] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 17:53:49,208 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676310859208 2023-02-13 17:53:49,208 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-13 17:53:49,208 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-13 17:53:49,208 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-13 17:53:49,209 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-13 17:53:49,209 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-13 17:53:49,209 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-13 17:53:49,209 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,210 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-13 17:53:49,210 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-13 17:53:49,211 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-13 17:53:49,211 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-13 17:53:49,211 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-13 17:53:49,211 INFO [RS:1;jenkins-hbase4:34875] regionserver.HRegionServer(952): ClusterId : 2a4478ba-ba99-441c-8812-a0c5f19f8b8a 2023-02-13 17:53:49,211 DEBUG [RS:1;jenkins-hbase4:34875] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 17:53:49,213 INFO [RS:2;jenkins-hbase4:42905] regionserver.HRegionServer(952): ClusterId : 2a4478ba-ba99-441c-8812-a0c5f19f8b8a 2023-02-13 17:53:49,213 DEBUG [RS:2;jenkins-hbase4:42905] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 17:53:49,213 DEBUG [RS:0;jenkins-hbase4:39461] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 17:53:49,213 DEBUG [RS:0;jenkins-hbase4:39461] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 17:53:49,213 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-13 17:53:49,213 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-13 17:53:49,213 DEBUG [RS:1;jenkins-hbase4:34875] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 17:53:49,213 DEBUG [RS:1;jenkins-hbase4:34875] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 17:53:49,215 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676310829211,5,FailOnTimeoutGroup] 2023-02-13 17:53:49,215 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-13 17:53:49,215 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676310829215,5,FailOnTimeoutGroup] 2023-02-13 17:53:49,215 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,215 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-13 17:53:49,215 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,215 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,217 DEBUG [RS:1;jenkins-hbase4:34875] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 17:53:49,218 DEBUG [RS:0;jenkins-hbase4:39461] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 17:53:49,218 DEBUG [RS:2;jenkins-hbase4:42905] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 17:53:49,218 DEBUG [RS:2;jenkins-hbase4:42905] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 17:53:49,219 DEBUG [RS:1;jenkins-hbase4:34875] zookeeper.ReadOnlyZKClient(139): Connect 0x72b79309 to 127.0.0.1:53630 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:53:49,219 DEBUG [RS:0;jenkins-hbase4:39461] zookeeper.ReadOnlyZKClient(139): Connect 0x741cd5b4 to 127.0.0.1:53630 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:53:49,226 DEBUG [RS:2;jenkins-hbase4:42905] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 17:53:49,229 DEBUG [RS:2;jenkins-hbase4:42905] zookeeper.ReadOnlyZKClient(139): Connect 0x20c49cc5 to 127.0.0.1:53630 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:53:49,244 DEBUG [RS:2;jenkins-hbase4:42905] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2e0376ad, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:53:49,245 DEBUG [RS:2;jenkins-hbase4:42905] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f3e7c96, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 17:53:49,245 DEBUG [RS:1;jenkins-hbase4:34875] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@784ff879, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:53:49,245 DEBUG [RS:1;jenkins-hbase4:34875] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d4d6914, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 17:53:49,245 DEBUG [RS:0;jenkins-hbase4:39461] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7adb3547, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:53:49,246 DEBUG [RS:0;jenkins-hbase4:39461] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7ee4bec4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 17:53:49,264 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-13 17:53:49,265 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-13 17:53:49,265 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e 2023-02-13 17:53:49,269 DEBUG [RS:2;jenkins-hbase4:42905] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:42905 2023-02-13 17:53:49,269 INFO [RS:2;jenkins-hbase4:42905] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 17:53:49,269 INFO [RS:2;jenkins-hbase4:42905] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 17:53:49,269 DEBUG [RS:2;jenkins-hbase4:42905] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 17:53:49,270 DEBUG [RS:1;jenkins-hbase4:34875] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:34875 2023-02-13 17:53:49,270 INFO [RS:1;jenkins-hbase4:34875] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 17:53:49,270 INFO [RS:1;jenkins-hbase4:34875] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 17:53:49,270 DEBUG [RS:1;jenkins-hbase4:34875] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 17:53:49,270 INFO [RS:2;jenkins-hbase4:42905] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,46373,1676310828881 with isa=jenkins-hbase4.apache.org/172.31.14.131:42905, startcode=1676310828981 2023-02-13 17:53:49,271 DEBUG [RS:2;jenkins-hbase4:42905] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 17:53:49,271 INFO [RS:1;jenkins-hbase4:34875] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,46373,1676310828881 with isa=jenkins-hbase4.apache.org/172.31.14.131:34875, startcode=1676310828951 2023-02-13 17:53:49,271 DEBUG [RS:1;jenkins-hbase4:34875] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 17:53:49,275 DEBUG [RS:0;jenkins-hbase4:39461] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:39461 2023-02-13 17:53:49,275 INFO [RS:0;jenkins-hbase4:39461] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 17:53:49,275 INFO [RS:0;jenkins-hbase4:39461] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 17:53:49,275 DEBUG [RS:0;jenkins-hbase4:39461] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 17:53:49,275 INFO [RS:0;jenkins-hbase4:39461] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,46373,1676310828881 with isa=jenkins-hbase4.apache.org/172.31.14.131:39461, startcode=1676310828923 2023-02-13 17:53:49,276 DEBUG [RS:0;jenkins-hbase4:39461] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 17:53:49,276 INFO [RS-EventLoopGroup-9-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60519, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.7 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 17:53:49,276 INFO [RS-EventLoopGroup-9-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46545, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.6 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 17:53:49,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46373] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,42905,1676310828981 2023-02-13 17:53:49,278 INFO [RS-EventLoopGroup-9-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46957, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.5 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 17:53:49,278 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46373] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,34875,1676310828951 2023-02-13 17:53:49,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46373] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,39461,1676310828923 2023-02-13 17:53:49,279 DEBUG [RS:2;jenkins-hbase4:42905] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e 2023-02-13 17:53:49,279 DEBUG [RS:1;jenkins-hbase4:34875] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e 2023-02-13 17:53:49,279 DEBUG [RS:1;jenkins-hbase4:34875] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:36959 2023-02-13 17:53:49,279 DEBUG [RS:1;jenkins-hbase4:34875] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 17:53:49,279 DEBUG [RS:2;jenkins-hbase4:42905] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:36959 2023-02-13 17:53:49,280 DEBUG [RS:0;jenkins-hbase4:39461] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e 2023-02-13 17:53:49,280 DEBUG [RS:2;jenkins-hbase4:42905] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 17:53:49,280 DEBUG [RS:0;jenkins-hbase4:39461] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:36959 2023-02-13 17:53:49,280 DEBUG [RS:0;jenkins-hbase4:39461] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 17:53:49,284 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:49,287 DEBUG [RS:1;jenkins-hbase4:34875] zookeeper.ZKUtil(162): regionserver:34875-0x1004ed140fc0002, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34875,1676310828951 2023-02-13 17:53:49,287 WARN [RS:1;jenkins-hbase4:34875] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 17:53:49,287 INFO [RS:1;jenkins-hbase4:34875] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 17:53:49,287 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,42905,1676310828981] 2023-02-13 17:53:49,288 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,39461,1676310828923] 2023-02-13 17:53:49,288 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,34875,1676310828951] 2023-02-13 17:53:49,288 DEBUG [RS:1;jenkins-hbase4:34875] regionserver.HRegionServer(1947): logDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,34875,1676310828951 2023-02-13 17:53:49,288 DEBUG [RS:0;jenkins-hbase4:39461] zookeeper.ZKUtil(162): regionserver:39461-0x1004ed140fc0001, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39461,1676310828923 2023-02-13 17:53:49,288 DEBUG [RS:2;jenkins-hbase4:42905] zookeeper.ZKUtil(162): regionserver:42905-0x1004ed140fc0003, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42905,1676310828981 2023-02-13 17:53:49,288 WARN [RS:0;jenkins-hbase4:39461] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 17:53:49,288 WARN [RS:2;jenkins-hbase4:42905] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 17:53:49,288 INFO [RS:0;jenkins-hbase4:39461] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 17:53:49,288 INFO [RS:2;jenkins-hbase4:42905] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 17:53:49,288 DEBUG [RS:0;jenkins-hbase4:39461] regionserver.HRegionServer(1947): logDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,39461,1676310828923 2023-02-13 17:53:49,288 DEBUG [RS:2;jenkins-hbase4:42905] regionserver.HRegionServer(1947): logDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,42905,1676310828981 2023-02-13 17:53:49,289 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:49,293 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 17:53:49,294 DEBUG [RS:1;jenkins-hbase4:34875] zookeeper.ZKUtil(162): regionserver:34875-0x1004ed140fc0002, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42905,1676310828981 2023-02-13 17:53:49,294 DEBUG [RS:0;jenkins-hbase4:39461] zookeeper.ZKUtil(162): regionserver:39461-0x1004ed140fc0001, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42905,1676310828981 2023-02-13 17:53:49,295 DEBUG [RS:0;jenkins-hbase4:39461] zookeeper.ZKUtil(162): regionserver:39461-0x1004ed140fc0001, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39461,1676310828923 2023-02-13 17:53:49,295 DEBUG [RS:1;jenkins-hbase4:34875] zookeeper.ZKUtil(162): regionserver:34875-0x1004ed140fc0002, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39461,1676310828923 2023-02-13 17:53:49,295 DEBUG [RS:0;jenkins-hbase4:39461] zookeeper.ZKUtil(162): regionserver:39461-0x1004ed140fc0001, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34875,1676310828951 2023-02-13 17:53:49,295 DEBUG [RS:1;jenkins-hbase4:34875] zookeeper.ZKUtil(162): regionserver:34875-0x1004ed140fc0002, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34875,1676310828951 2023-02-13 17:53:49,296 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/info 2023-02-13 17:53:49,296 DEBUG [RS:2;jenkins-hbase4:42905] zookeeper.ZKUtil(162): regionserver:42905-0x1004ed140fc0003, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42905,1676310828981 2023-02-13 17:53:49,296 DEBUG [RS:2;jenkins-hbase4:42905] zookeeper.ZKUtil(162): regionserver:42905-0x1004ed140fc0003, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39461,1676310828923 2023-02-13 17:53:49,296 DEBUG [RS:1;jenkins-hbase4:34875] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 17:53:49,296 DEBUG [RS:0;jenkins-hbase4:39461] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 17:53:49,297 DEBUG [RS:2;jenkins-hbase4:42905] zookeeper.ZKUtil(162): regionserver:42905-0x1004ed140fc0003, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34875,1676310828951 2023-02-13 17:53:49,297 INFO [RS:0;jenkins-hbase4:39461] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 17:53:49,297 INFO [RS:1;jenkins-hbase4:34875] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 17:53:49,297 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 17:53:49,297 DEBUG [RS:2;jenkins-hbase4:42905] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 17:53:49,298 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:49,298 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 17:53:49,299 INFO [RS:2;jenkins-hbase4:42905] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 17:53:49,299 INFO [RS:0;jenkins-hbase4:39461] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 17:53:49,300 INFO [RS:0;jenkins-hbase4:39461] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 17:53:49,300 INFO [RS:0;jenkins-hbase4:39461] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,300 INFO [RS:1;jenkins-hbase4:34875] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 17:53:49,300 INFO [RS:0;jenkins-hbase4:39461] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 17:53:49,301 INFO [RS:1;jenkins-hbase4:34875] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 17:53:49,301 INFO [RS:1;jenkins-hbase4:34875] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,304 INFO [RS:1;jenkins-hbase4:34875] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 17:53:49,308 INFO [RS:0;jenkins-hbase4:39461] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,309 INFO [RS:2;jenkins-hbase4:42905] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 17:53:49,309 DEBUG [RS:0;jenkins-hbase4:39461] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,309 DEBUG [RS:0;jenkins-hbase4:39461] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,309 DEBUG [RS:0;jenkins-hbase4:39461] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,309 INFO [RS:2;jenkins-hbase4:42905] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 17:53:49,309 DEBUG [RS:0;jenkins-hbase4:39461] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,309 INFO [RS:2;jenkins-hbase4:42905] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,309 DEBUG [RS:0;jenkins-hbase4:39461] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,309 INFO [RS:1;jenkins-hbase4:34875] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,309 DEBUG [RS:0;jenkins-hbase4:39461] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 17:53:49,309 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/rep_barrier 2023-02-13 17:53:49,309 DEBUG [RS:0;jenkins-hbase4:39461] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,310 DEBUG [RS:0;jenkins-hbase4:39461] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,310 DEBUG [RS:0;jenkins-hbase4:39461] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,310 DEBUG [RS:0;jenkins-hbase4:39461] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,310 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 17:53:49,310 INFO [RS:2;jenkins-hbase4:42905] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 17:53:49,312 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:49,312 DEBUG [RS:1;jenkins-hbase4:34875] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,312 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 17:53:49,312 DEBUG [RS:1;jenkins-hbase4:34875] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,312 DEBUG [RS:1;jenkins-hbase4:34875] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,312 DEBUG [RS:1;jenkins-hbase4:34875] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,312 DEBUG [RS:1;jenkins-hbase4:34875] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,312 DEBUG [RS:1;jenkins-hbase4:34875] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 17:53:49,312 DEBUG [RS:1;jenkins-hbase4:34875] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,313 DEBUG [RS:1;jenkins-hbase4:34875] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,313 DEBUG [RS:1;jenkins-hbase4:34875] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,313 DEBUG [RS:1;jenkins-hbase4:34875] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,314 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/table 2023-02-13 17:53:49,314 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 17:53:49,315 INFO [RS:1;jenkins-hbase4:34875] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,315 INFO [RS:1;jenkins-hbase4:34875] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,315 INFO [RS:1;jenkins-hbase4:34875] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,315 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:49,320 INFO [RS:2;jenkins-hbase4:42905] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,320 INFO [RS:0;jenkins-hbase4:39461] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,320 DEBUG [RS:2;jenkins-hbase4:42905] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,320 INFO [RS:0;jenkins-hbase4:39461] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,320 DEBUG [RS:2;jenkins-hbase4:42905] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,320 INFO [RS:0;jenkins-hbase4:39461] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,320 DEBUG [RS:2;jenkins-hbase4:42905] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,320 DEBUG [RS:2;jenkins-hbase4:42905] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,320 DEBUG [RS:2;jenkins-hbase4:42905] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,320 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740 2023-02-13 17:53:49,320 DEBUG [RS:2;jenkins-hbase4:42905] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 17:53:49,321 DEBUG [RS:2;jenkins-hbase4:42905] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,321 DEBUG [RS:2;jenkins-hbase4:42905] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,321 DEBUG [RS:2;jenkins-hbase4:42905] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,321 DEBUG [RS:2;jenkins-hbase4:42905] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:49,321 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740 2023-02-13 17:53:49,324 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 17:53:49,325 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 17:53:49,333 INFO [RS:2;jenkins-hbase4:42905] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,333 INFO [RS:2;jenkins-hbase4:42905] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,333 INFO [RS:2;jenkins-hbase4:42905] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,333 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:53:49,334 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67723286, jitterRate=0.009155601263046265}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 17:53:49,334 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1676310829289Initializing all the Stores at 1676310829293 (+4 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676310829293Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676310829293Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676310829293Cleaning up temporary data from old regions at 1676310829322 (+29 ms)Cleaning up detritus from prior splits at 1676310829323 (+1 ms)Region opened successfully at 1676310829334 (+11 ms) 2023-02-13 17:53:49,334 INFO [RS:1;jenkins-hbase4:34875] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 17:53:49,334 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 17:53:49,335 INFO [RS:1;jenkins-hbase4:34875] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34875,1676310828951-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,335 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 17:53:49,335 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 17:53:49,335 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 17:53:49,335 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 17:53:49,336 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 17:53:49,336 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676310829334Disabling compacts and flushes for region at 1676310829334Disabling writes for close at 1676310829335 (+1 ms)Writing region close event to WAL at 1676310829336 (+1 ms)Closed at 1676310829336 2023-02-13 17:53:49,340 INFO [RS:0;jenkins-hbase4:39461] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 17:53:49,340 INFO [RS:0;jenkins-hbase4:39461] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39461,1676310828923-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,344 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-13 17:53:49,344 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-13 17:53:49,344 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-13 17:53:49,347 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-13 17:53:49,349 INFO [RS:2;jenkins-hbase4:42905] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 17:53:49,349 INFO [RS:2;jenkins-hbase4:42905] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42905,1676310828981-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,350 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-13 17:53:49,352 INFO [RS:1;jenkins-hbase4:34875] regionserver.Replication(203): jenkins-hbase4.apache.org,34875,1676310828951 started 2023-02-13 17:53:49,352 INFO [RS:1;jenkins-hbase4:34875] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,34875,1676310828951, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:34875, sessionid=0x1004ed140fc0002 2023-02-13 17:53:49,352 DEBUG [RS:1;jenkins-hbase4:34875] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 17:53:49,352 DEBUG [RS:1;jenkins-hbase4:34875] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,34875,1676310828951 2023-02-13 17:53:49,352 DEBUG [RS:1;jenkins-hbase4:34875] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34875,1676310828951' 2023-02-13 17:53:49,352 DEBUG [RS:1;jenkins-hbase4:34875] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 17:53:49,354 DEBUG [RS:1;jenkins-hbase4:34875] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 17:53:49,355 DEBUG [RS:1;jenkins-hbase4:34875] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 17:53:49,355 DEBUG [RS:1;jenkins-hbase4:34875] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 17:53:49,355 DEBUG [RS:1;jenkins-hbase4:34875] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,34875,1676310828951 2023-02-13 17:53:49,355 DEBUG [RS:1;jenkins-hbase4:34875] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34875,1676310828951' 2023-02-13 17:53:49,355 DEBUG [RS:1;jenkins-hbase4:34875] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 17:53:49,355 DEBUG [RS:1;jenkins-hbase4:34875] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 17:53:49,356 DEBUG [RS:1;jenkins-hbase4:34875] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 17:53:49,356 INFO [RS:1;jenkins-hbase4:34875] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 17:53:49,356 INFO [RS:1;jenkins-hbase4:34875] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 17:53:49,359 INFO [RS:0;jenkins-hbase4:39461] regionserver.Replication(203): jenkins-hbase4.apache.org,39461,1676310828923 started 2023-02-13 17:53:49,359 INFO [RS:0;jenkins-hbase4:39461] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,39461,1676310828923, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:39461, sessionid=0x1004ed140fc0001 2023-02-13 17:53:49,360 DEBUG [RS:0;jenkins-hbase4:39461] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 17:53:49,360 DEBUG [RS:0;jenkins-hbase4:39461] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,39461,1676310828923 2023-02-13 17:53:49,360 DEBUG [RS:0;jenkins-hbase4:39461] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39461,1676310828923' 2023-02-13 17:53:49,360 DEBUG [RS:0;jenkins-hbase4:39461] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 17:53:49,360 DEBUG [RS:0;jenkins-hbase4:39461] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 17:53:49,360 DEBUG [RS:0;jenkins-hbase4:39461] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 17:53:49,360 DEBUG [RS:0;jenkins-hbase4:39461] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 17:53:49,360 DEBUG [RS:0;jenkins-hbase4:39461] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,39461,1676310828923 2023-02-13 17:53:49,360 DEBUG [RS:0;jenkins-hbase4:39461] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39461,1676310828923' 2023-02-13 17:53:49,361 DEBUG [RS:0;jenkins-hbase4:39461] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 17:53:49,361 DEBUG [RS:0;jenkins-hbase4:39461] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 17:53:49,361 DEBUG [RS:0;jenkins-hbase4:39461] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 17:53:49,361 INFO [RS:0;jenkins-hbase4:39461] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 17:53:49,361 INFO [RS:0;jenkins-hbase4:39461] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 17:53:49,363 INFO [RS:2;jenkins-hbase4:42905] regionserver.Replication(203): jenkins-hbase4.apache.org,42905,1676310828981 started 2023-02-13 17:53:49,363 INFO [RS:2;jenkins-hbase4:42905] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,42905,1676310828981, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:42905, sessionid=0x1004ed140fc0003 2023-02-13 17:53:49,363 DEBUG [RS:2;jenkins-hbase4:42905] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 17:53:49,363 DEBUG [RS:2;jenkins-hbase4:42905] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,42905,1676310828981 2023-02-13 17:53:49,363 DEBUG [RS:2;jenkins-hbase4:42905] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42905,1676310828981' 2023-02-13 17:53:49,363 DEBUG [RS:2;jenkins-hbase4:42905] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 17:53:49,363 DEBUG [RS:2;jenkins-hbase4:42905] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 17:53:49,364 DEBUG [RS:2;jenkins-hbase4:42905] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 17:53:49,364 DEBUG [RS:2;jenkins-hbase4:42905] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 17:53:49,364 DEBUG [RS:2;jenkins-hbase4:42905] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,42905,1676310828981 2023-02-13 17:53:49,364 DEBUG [RS:2;jenkins-hbase4:42905] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42905,1676310828981' 2023-02-13 17:53:49,364 DEBUG [RS:2;jenkins-hbase4:42905] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 17:53:49,364 DEBUG [RS:2;jenkins-hbase4:42905] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 17:53:49,364 DEBUG [RS:2;jenkins-hbase4:42905] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 17:53:49,364 INFO [RS:2;jenkins-hbase4:42905] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 17:53:49,364 INFO [RS:2;jenkins-hbase4:42905] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 17:53:49,458 INFO [RS:1;jenkins-hbase4:34875] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34875%2C1676310828951, suffix=, logDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,34875,1676310828951, archiveDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/oldWALs, maxLogs=32 2023-02-13 17:53:49,463 INFO [RS:0;jenkins-hbase4:39461] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39461%2C1676310828923, suffix=, logDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,39461,1676310828923, archiveDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/oldWALs, maxLogs=32 2023-02-13 17:53:49,466 INFO [RS:2;jenkins-hbase4:42905] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42905%2C1676310828981, suffix=, logDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,42905,1676310828981, archiveDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/oldWALs, maxLogs=32 2023-02-13 17:53:49,478 DEBUG [RS-EventLoopGroup-12-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39523,DS-cf1e0e36-9598-4b66-ab37-608d409fcf08,DISK] 2023-02-13 17:53:49,480 DEBUG [RS-EventLoopGroup-12-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37727,DS-c79d1410-43a1-4b12-a2c2-dff1472abceb,DISK] 2023-02-13 17:53:49,480 DEBUG [RS-EventLoopGroup-12-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37061,DS-c817cfbe-f58e-4f4d-94e0-14281a32ca4c,DISK] 2023-02-13 17:53:49,489 DEBUG [RS-EventLoopGroup-12-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39523,DS-cf1e0e36-9598-4b66-ab37-608d409fcf08,DISK] 2023-02-13 17:53:49,489 DEBUG [RS-EventLoopGroup-12-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37727,DS-c79d1410-43a1-4b12-a2c2-dff1472abceb,DISK] 2023-02-13 17:53:49,489 DEBUG [RS-EventLoopGroup-12-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39523,DS-cf1e0e36-9598-4b66-ab37-608d409fcf08,DISK] 2023-02-13 17:53:49,490 DEBUG [RS-EventLoopGroup-12-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37061,DS-c817cfbe-f58e-4f4d-94e0-14281a32ca4c,DISK] 2023-02-13 17:53:49,490 DEBUG [RS-EventLoopGroup-12-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37061,DS-c817cfbe-f58e-4f4d-94e0-14281a32ca4c,DISK] 2023-02-13 17:53:49,490 DEBUG [RS-EventLoopGroup-12-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37727,DS-c79d1410-43a1-4b12-a2c2-dff1472abceb,DISK] 2023-02-13 17:53:49,490 INFO [RS:1;jenkins-hbase4:34875] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,34875,1676310828951/jenkins-hbase4.apache.org%2C34875%2C1676310828951.1676310829459 2023-02-13 17:53:49,491 DEBUG [RS:1;jenkins-hbase4:34875] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37727,DS-c79d1410-43a1-4b12-a2c2-dff1472abceb,DISK], DatanodeInfoWithStorage[127.0.0.1:39523,DS-cf1e0e36-9598-4b66-ab37-608d409fcf08,DISK], DatanodeInfoWithStorage[127.0.0.1:37061,DS-c817cfbe-f58e-4f4d-94e0-14281a32ca4c,DISK]] 2023-02-13 17:53:49,492 INFO [RS:0;jenkins-hbase4:39461] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,39461,1676310828923/jenkins-hbase4.apache.org%2C39461%2C1676310828923.1676310829464 2023-02-13 17:53:49,492 DEBUG [RS:0;jenkins-hbase4:39461] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39523,DS-cf1e0e36-9598-4b66-ab37-608d409fcf08,DISK], DatanodeInfoWithStorage[127.0.0.1:37061,DS-c817cfbe-f58e-4f4d-94e0-14281a32ca4c,DISK], DatanodeInfoWithStorage[127.0.0.1:37727,DS-c79d1410-43a1-4b12-a2c2-dff1472abceb,DISK]] 2023-02-13 17:53:49,493 INFO [RS:2;jenkins-hbase4:42905] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,42905,1676310828981/jenkins-hbase4.apache.org%2C42905%2C1676310828981.1676310829467 2023-02-13 17:53:49,493 DEBUG [RS:2;jenkins-hbase4:42905] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37727,DS-c79d1410-43a1-4b12-a2c2-dff1472abceb,DISK], DatanodeInfoWithStorage[127.0.0.1:37061,DS-c817cfbe-f58e-4f4d-94e0-14281a32ca4c,DISK], DatanodeInfoWithStorage[127.0.0.1:39523,DS-cf1e0e36-9598-4b66-ab37-608d409fcf08,DISK]] 2023-02-13 17:53:49,502 DEBUG [jenkins-hbase4:46373] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-13 17:53:49,502 DEBUG [jenkins-hbase4:46373] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 17:53:49,504 DEBUG [jenkins-hbase4:46373] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 17:53:49,504 DEBUG [jenkins-hbase4:46373] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 17:53:49,504 DEBUG [jenkins-hbase4:46373] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 17:53:49,504 DEBUG [jenkins-hbase4:46373] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 17:53:49,505 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,42905,1676310828981, state=OPENING 2023-02-13 17:53:49,506 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-13 17:53:49,509 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:53:49,509 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,42905,1676310828981}] 2023-02-13 17:53:49,509 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 17:53:49,663 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,42905,1676310828981 2023-02-13 17:53:49,663 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 17:53:49,665 INFO [RS-EventLoopGroup-12-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52602, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 17:53:49,669 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-13 17:53:49,669 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 17:53:49,671 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42905%2C1676310828981.meta, suffix=.meta, logDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,42905,1676310828981, archiveDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/oldWALs, maxLogs=32 2023-02-13 17:53:49,685 DEBUG [RS-EventLoopGroup-12-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39523,DS-cf1e0e36-9598-4b66-ab37-608d409fcf08,DISK] 2023-02-13 17:53:49,685 DEBUG [RS-EventLoopGroup-12-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37727,DS-c79d1410-43a1-4b12-a2c2-dff1472abceb,DISK] 2023-02-13 17:53:49,685 DEBUG [RS-EventLoopGroup-12-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37061,DS-c817cfbe-f58e-4f4d-94e0-14281a32ca4c,DISK] 2023-02-13 17:53:49,687 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,42905,1676310828981/jenkins-hbase4.apache.org%2C42905%2C1676310828981.meta.1676310829672.meta 2023-02-13 17:53:49,687 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39523,DS-cf1e0e36-9598-4b66-ab37-608d409fcf08,DISK], DatanodeInfoWithStorage[127.0.0.1:37061,DS-c817cfbe-f58e-4f4d-94e0-14281a32ca4c,DISK], DatanodeInfoWithStorage[127.0.0.1:37727,DS-c79d1410-43a1-4b12-a2c2-dff1472abceb,DISK]] 2023-02-13 17:53:49,687 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-13 17:53:49,688 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-13 17:53:49,688 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-13 17:53:49,688 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-13 17:53:49,688 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-13 17:53:49,688 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:49,688 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-13 17:53:49,688 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-13 17:53:49,690 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 17:53:49,691 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/info 2023-02-13 17:53:49,691 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/info 2023-02-13 17:53:49,691 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 17:53:49,692 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:49,692 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 17:53:49,693 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/rep_barrier 2023-02-13 17:53:49,693 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/rep_barrier 2023-02-13 17:53:49,693 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 17:53:49,694 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:49,694 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 17:53:49,695 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/table 2023-02-13 17:53:49,695 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/table 2023-02-13 17:53:49,696 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 17:53:49,696 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:49,697 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740 2023-02-13 17:53:49,698 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740 2023-02-13 17:53:49,701 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 17:53:49,702 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 17:53:49,703 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73588136, jitterRate=0.09654867649078369}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 17:53:49,703 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676310829688Writing region info on filesystem at 1676310829688Initializing all the Stores at 1676310829689 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676310829689Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676310829689Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676310829689Cleaning up temporary data from old regions at 1676310829699 (+10 ms)Cleaning up detritus from prior splits at 1676310829700 (+1 ms)Running coprocessor post-open hooks at 1676310829703 (+3 ms)Region opened successfully at 1676310829703 2023-02-13 17:53:49,704 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676310829663 2023-02-13 17:53:49,708 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-13 17:53:49,708 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-13 17:53:49,709 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,42905,1676310828981, state=OPEN 2023-02-13 17:53:49,712 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-13 17:53:49,712 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 17:53:49,714 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-13 17:53:49,714 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,42905,1676310828981 in 203 msec 2023-02-13 17:53:49,717 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-13 17:53:49,717 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 370 msec 2023-02-13 17:53:49,719 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 514 msec 2023-02-13 17:53:49,719 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676310829008, completionTime=-1 2023-02-13 17:53:49,719 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-13 17:53:49,719 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-13 17:53:49,722 DEBUG [hconnection-0x7ba1f5e3-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:53:49,723 INFO [RS-EventLoopGroup-12-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52610, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:53:49,725 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-13 17:53:49,725 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676310889725 2023-02-13 17:53:49,725 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676310949725 2023-02-13 17:53:49,725 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 5 msec 2023-02-13 17:53:49,732 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46373,1676310828881-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,732 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46373,1676310828881-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,732 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46373,1676310828881-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,732 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:46373, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,732 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:49,732 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-13 17:53:49,732 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-13 17:53:49,733 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-13 17:53:49,734 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-13 17:53:49,735 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 17:53:49,736 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 17:53:49,738 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/.tmp/data/hbase/namespace/28d66b4066b494c8b8644327cd7fdb67 2023-02-13 17:53:49,739 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/.tmp/data/hbase/namespace/28d66b4066b494c8b8644327cd7fdb67 empty. 2023-02-13 17:53:49,739 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/.tmp/data/hbase/namespace/28d66b4066b494c8b8644327cd7fdb67 2023-02-13 17:53:49,739 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-13 17:53:49,755 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-13 17:53:49,756 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => 28d66b4066b494c8b8644327cd7fdb67, NAME => 'hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/.tmp 2023-02-13 17:53:49,768 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:49,768 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing 28d66b4066b494c8b8644327cd7fdb67, disabling compactions & flushes 2023-02-13 17:53:49,768 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67. 2023-02-13 17:53:49,768 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67. 2023-02-13 17:53:49,769 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67. after waiting 0 ms 2023-02-13 17:53:49,769 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67. 2023-02-13 17:53:49,769 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67. 2023-02-13 17:53:49,769 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for 28d66b4066b494c8b8644327cd7fdb67: Waiting for close lock at 1676310829768Disabling compacts and flushes for region at 1676310829768Disabling writes for close at 1676310829769 (+1 ms)Writing region close event to WAL at 1676310829769Closed at 1676310829769 2023-02-13 17:53:49,771 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 17:53:49,772 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676310829772"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676310829772"}]},"ts":"1676310829772"} 2023-02-13 17:53:49,775 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 17:53:49,776 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 17:53:49,776 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676310829776"}]},"ts":"1676310829776"} 2023-02-13 17:53:49,778 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-13 17:53:49,782 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 17:53:49,782 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 17:53:49,783 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 17:53:49,783 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 17:53:49,783 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 17:53:49,783 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=28d66b4066b494c8b8644327cd7fdb67, ASSIGN}] 2023-02-13 17:53:49,785 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=28d66b4066b494c8b8644327cd7fdb67, ASSIGN 2023-02-13 17:53:49,786 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=28d66b4066b494c8b8644327cd7fdb67, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34875,1676310828951; forceNewPlan=false, retain=false 2023-02-13 17:53:49,937 INFO [jenkins-hbase4:46373] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 17:53:49,938 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=28d66b4066b494c8b8644327cd7fdb67, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34875,1676310828951 2023-02-13 17:53:49,938 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676310829938"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676310829938"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676310829938"}]},"ts":"1676310829938"} 2023-02-13 17:53:49,940 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 28d66b4066b494c8b8644327cd7fdb67, server=jenkins-hbase4.apache.org,34875,1676310828951}] 2023-02-13 17:53:50,093 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,34875,1676310828951 2023-02-13 17:53:50,093 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 17:53:50,095 INFO [RS-EventLoopGroup-11-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44146, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 17:53:50,099 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67. 2023-02-13 17:53:50,099 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 28d66b4066b494c8b8644327cd7fdb67, NAME => 'hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67.', STARTKEY => '', ENDKEY => ''} 2023-02-13 17:53:50,099 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 28d66b4066b494c8b8644327cd7fdb67 2023-02-13 17:53:50,100 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:50,100 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 28d66b4066b494c8b8644327cd7fdb67 2023-02-13 17:53:50,100 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 28d66b4066b494c8b8644327cd7fdb67 2023-02-13 17:53:50,101 INFO [StoreOpener-28d66b4066b494c8b8644327cd7fdb67-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 28d66b4066b494c8b8644327cd7fdb67 2023-02-13 17:53:50,102 DEBUG [StoreOpener-28d66b4066b494c8b8644327cd7fdb67-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/namespace/28d66b4066b494c8b8644327cd7fdb67/info 2023-02-13 17:53:50,102 DEBUG [StoreOpener-28d66b4066b494c8b8644327cd7fdb67-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/namespace/28d66b4066b494c8b8644327cd7fdb67/info 2023-02-13 17:53:50,103 INFO [StoreOpener-28d66b4066b494c8b8644327cd7fdb67-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 28d66b4066b494c8b8644327cd7fdb67 columnFamilyName info 2023-02-13 17:53:50,103 INFO [StoreOpener-28d66b4066b494c8b8644327cd7fdb67-1] regionserver.HStore(310): Store=28d66b4066b494c8b8644327cd7fdb67/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:50,104 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/namespace/28d66b4066b494c8b8644327cd7fdb67 2023-02-13 17:53:50,105 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/namespace/28d66b4066b494c8b8644327cd7fdb67 2023-02-13 17:53:50,107 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 28d66b4066b494c8b8644327cd7fdb67 2023-02-13 17:53:50,110 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/namespace/28d66b4066b494c8b8644327cd7fdb67/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:53:50,110 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 28d66b4066b494c8b8644327cd7fdb67; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61375892, jitterRate=-0.08542793989181519}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:53:50,110 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 28d66b4066b494c8b8644327cd7fdb67: Running coprocessor pre-open hook at 1676310830100Writing region info on filesystem at 1676310830100Initializing all the Stores at 1676310830101 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676310830101Cleaning up temporary data from old regions at 1676310830105 (+4 ms)Cleaning up detritus from prior splits at 1676310830105Running coprocessor post-open hooks at 1676310830110 (+5 ms)Region opened successfully at 1676310830110 2023-02-13 17:53:50,111 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67., pid=6, masterSystemTime=1676310830093 2023-02-13 17:53:50,114 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67. 2023-02-13 17:53:50,115 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67. 2023-02-13 17:53:50,115 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=28d66b4066b494c8b8644327cd7fdb67, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34875,1676310828951 2023-02-13 17:53:50,116 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676310830115"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676310830115"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676310830115"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676310830115"}]},"ts":"1676310830115"} 2023-02-13 17:53:50,119 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-13 17:53:50,119 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 28d66b4066b494c8b8644327cd7fdb67, server=jenkins-hbase4.apache.org,34875,1676310828951 in 177 msec 2023-02-13 17:53:50,122 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-13 17:53:50,122 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=28d66b4066b494c8b8644327cd7fdb67, ASSIGN in 336 msec 2023-02-13 17:53:50,123 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 17:53:50,123 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676310830123"}]},"ts":"1676310830123"} 2023-02-13 17:53:50,124 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-13 17:53:50,127 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 17:53:50,128 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 395 msec 2023-02-13 17:53:50,134 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-13 17:53:50,135 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-13 17:53:50,135 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:53:50,138 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:53:50,139 INFO [RS-EventLoopGroup-11-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44154, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:53:50,141 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-13 17:53:50,151 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-13 17:53:50,154 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 13 msec 2023-02-13 17:53:50,163 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-13 17:53:50,170 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-13 17:53:50,173 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 10 msec 2023-02-13 17:53:50,187 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-13 17:53:50,189 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-13 17:53:50,189 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 1.172sec 2023-02-13 17:53:50,190 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-13 17:53:50,190 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-13 17:53:50,190 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-13 17:53:50,190 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46373,1676310828881-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-13 17:53:50,190 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46373,1676310828881-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-13 17:53:50,191 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-13 17:53:50,210 DEBUG [Listener at localhost/40181] zookeeper.ReadOnlyZKClient(139): Connect 0x410494f3 to 127.0.0.1:53630 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:53:50,215 DEBUG [Listener at localhost/40181] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@26e673a4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:53:50,217 DEBUG [hconnection-0x4190e4f6-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:53:50,219 INFO [RS-EventLoopGroup-12-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52620, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:53:50,220 INFO [Listener at localhost/40181] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,46373,1676310828881 2023-02-13 17:53:50,222 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/balancer 2023-02-13 17:53:50,222 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:53:50,223 INFO [Listener at localhost/40181] master.MasterRpcServices(492): Client=null/null set balanceSwitch=false 2023-02-13 17:53:50,223 INFO [Listener at localhost/40181] master.TestRetainAssignmentOnRestart(212): Creating tables 2023-02-13 17:53:50,225 DEBUG [Listener at localhost/40181] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-13 17:53:50,226 INFO [RS-EventLoopGroup-9-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33772, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-13 17:53:50,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46373] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'restartTableOne', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 17:53:50,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46373] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableOne 2023-02-13 17:53:50,232 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 17:53:50,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46373] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "restartTableOne" procId is: 9 2023-02-13 17:53:50,233 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 17:53:50,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46373] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 17:53:50,235 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/.tmp/data/default/restartTableOne/fbc323e3b53470e4b5a74c9688484c26 2023-02-13 17:53:50,236 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/.tmp/data/default/restartTableOne/fbc323e3b53470e4b5a74c9688484c26 empty. 2023-02-13 17:53:50,236 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/.tmp/data/default/restartTableOne/fbc323e3b53470e4b5a74c9688484c26 2023-02-13 17:53:50,236 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived restartTableOne regions 2023-02-13 17:53:50,251 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/.tmp/data/default/restartTableOne/.tabledesc/.tableinfo.0000000001 2023-02-13 17:53:50,252 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(7671): creating {ENCODED => fbc323e3b53470e4b5a74c9688484c26, NAME => 'restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableOne', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/.tmp 2023-02-13 17:53:50,271 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:50,271 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1603): Closing fbc323e3b53470e4b5a74c9688484c26, disabling compactions & flushes 2023-02-13 17:53:50,271 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26. 2023-02-13 17:53:50,271 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26. 2023-02-13 17:53:50,271 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26. after waiting 0 ms 2023-02-13 17:53:50,271 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26. 2023-02-13 17:53:50,271 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1837): Closed restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26. 2023-02-13 17:53:50,272 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1557): Region close journal for fbc323e3b53470e4b5a74c9688484c26: Waiting for close lock at 1676310830271Disabling compacts and flushes for region at 1676310830271Disabling writes for close at 1676310830271Writing region close event to WAL at 1676310830271Closed at 1676310830271 2023-02-13 17:53:50,275 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 17:53:50,276 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676310830276"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676310830276"}]},"ts":"1676310830276"} 2023-02-13 17:53:50,278 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 17:53:50,279 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 17:53:50,280 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableOne","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676310830280"}]},"ts":"1676310830280"} 2023-02-13 17:53:50,281 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=restartTableOne, state=ENABLING in hbase:meta 2023-02-13 17:53:50,285 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 17:53:50,286 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 17:53:50,286 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 17:53:50,286 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 17:53:50,286 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 17:53:50,286 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=fbc323e3b53470e4b5a74c9688484c26, ASSIGN}] 2023-02-13 17:53:50,288 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=fbc323e3b53470e4b5a74c9688484c26, ASSIGN 2023-02-13 17:53:50,289 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableOne, region=fbc323e3b53470e4b5a74c9688484c26, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42905,1676310828981; forceNewPlan=false, retain=false 2023-02-13 17:53:50,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46373] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 17:53:50,440 INFO [jenkins-hbase4:46373] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 17:53:50,441 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=fbc323e3b53470e4b5a74c9688484c26, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42905,1676310828981 2023-02-13 17:53:50,441 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676310830441"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676310830441"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676310830441"}]},"ts":"1676310830441"} 2023-02-13 17:53:50,443 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure fbc323e3b53470e4b5a74c9688484c26, server=jenkins-hbase4.apache.org,42905,1676310828981}] 2023-02-13 17:53:50,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46373] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 17:53:50,599 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26. 2023-02-13 17:53:50,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => fbc323e3b53470e4b5a74c9688484c26, NAME => 'restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26.', STARTKEY => '', ENDKEY => ''} 2023-02-13 17:53:50,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableOne fbc323e3b53470e4b5a74c9688484c26 2023-02-13 17:53:50,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:50,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for fbc323e3b53470e4b5a74c9688484c26 2023-02-13 17:53:50,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for fbc323e3b53470e4b5a74c9688484c26 2023-02-13 17:53:50,602 INFO [StoreOpener-fbc323e3b53470e4b5a74c9688484c26-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region fbc323e3b53470e4b5a74c9688484c26 2023-02-13 17:53:50,603 DEBUG [StoreOpener-fbc323e3b53470e4b5a74c9688484c26-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableOne/fbc323e3b53470e4b5a74c9688484c26/family 2023-02-13 17:53:50,603 DEBUG [StoreOpener-fbc323e3b53470e4b5a74c9688484c26-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableOne/fbc323e3b53470e4b5a74c9688484c26/family 2023-02-13 17:53:50,604 INFO [StoreOpener-fbc323e3b53470e4b5a74c9688484c26-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fbc323e3b53470e4b5a74c9688484c26 columnFamilyName family 2023-02-13 17:53:50,605 INFO [StoreOpener-fbc323e3b53470e4b5a74c9688484c26-1] regionserver.HStore(310): Store=fbc323e3b53470e4b5a74c9688484c26/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:50,606 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableOne/fbc323e3b53470e4b5a74c9688484c26 2023-02-13 17:53:50,607 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableOne/fbc323e3b53470e4b5a74c9688484c26 2023-02-13 17:53:50,610 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for fbc323e3b53470e4b5a74c9688484c26 2023-02-13 17:53:50,612 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableOne/fbc323e3b53470e4b5a74c9688484c26/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:53:50,613 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened fbc323e3b53470e4b5a74c9688484c26; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65206121, jitterRate=-0.02835308015346527}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:53:50,613 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for fbc323e3b53470e4b5a74c9688484c26: Running coprocessor pre-open hook at 1676310830600Writing region info on filesystem at 1676310830600Initializing all the Stores at 1676310830601 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676310830601Cleaning up temporary data from old regions at 1676310830607 (+6 ms)Cleaning up detritus from prior splits at 1676310830608 (+1 ms)Running coprocessor post-open hooks at 1676310830613 (+5 ms)Region opened successfully at 1676310830613 2023-02-13 17:53:50,614 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26., pid=11, masterSystemTime=1676310830595 2023-02-13 17:53:50,616 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26. 2023-02-13 17:53:50,616 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26. 2023-02-13 17:53:50,617 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=fbc323e3b53470e4b5a74c9688484c26, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42905,1676310828981 2023-02-13 17:53:50,617 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676310830616"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676310830616"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676310830616"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676310830616"}]},"ts":"1676310830616"} 2023-02-13 17:53:50,621 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-02-13 17:53:50,622 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure fbc323e3b53470e4b5a74c9688484c26, server=jenkins-hbase4.apache.org,42905,1676310828981 in 176 msec 2023-02-13 17:53:50,624 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-02-13 17:53:50,624 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=restartTableOne, region=fbc323e3b53470e4b5a74c9688484c26, ASSIGN in 336 msec 2023-02-13 17:53:50,625 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 17:53:50,625 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableOne","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676310830625"}]},"ts":"1676310830625"} 2023-02-13 17:53:50,627 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=restartTableOne, state=ENABLED in hbase:meta 2023-02-13 17:53:50,629 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 17:53:50,631 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=restartTableOne in 401 msec 2023-02-13 17:53:50,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46373] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 17:53:50,838 INFO [Listener at localhost/40181] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableOne, procId: 9 completed 2023-02-13 17:53:50,838 DEBUG [Listener at localhost/40181] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableOne get assigned. Timeout = 60000ms 2023-02-13 17:53:50,838 INFO [Listener at localhost/40181] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:50,841 INFO [Listener at localhost/40181] hbase.HBaseTestingUtility(3484): All regions for table restartTableOne assigned to meta. Checking AM states. 2023-02-13 17:53:50,841 INFO [Listener at localhost/40181] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:50,841 INFO [Listener at localhost/40181] hbase.HBaseTestingUtility(3504): All regions for table restartTableOne assigned. 2023-02-13 17:53:50,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46373] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'restartTableTwo', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 17:53:50,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46373] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableTwo 2023-02-13 17:53:50,847 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 17:53:50,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46373] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "restartTableTwo" procId is: 12 2023-02-13 17:53:50,848 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 17:53:50,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46373] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 17:53:50,850 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/.tmp/data/default/restartTableTwo/52ceb95a47ba04015d52c77bc1874032 2023-02-13 17:53:50,851 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/.tmp/data/default/restartTableTwo/52ceb95a47ba04015d52c77bc1874032 empty. 2023-02-13 17:53:50,852 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/.tmp/data/default/restartTableTwo/52ceb95a47ba04015d52c77bc1874032 2023-02-13 17:53:50,852 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived restartTableTwo regions 2023-02-13 17:53:50,872 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/.tmp/data/default/restartTableTwo/.tabledesc/.tableinfo.0000000001 2023-02-13 17:53:50,874 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(7671): creating {ENCODED => 52ceb95a47ba04015d52c77bc1874032, NAME => 'restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableTwo', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/.tmp 2023-02-13 17:53:50,893 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:50,893 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1603): Closing 52ceb95a47ba04015d52c77bc1874032, disabling compactions & flushes 2023-02-13 17:53:50,893 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032. 2023-02-13 17:53:50,893 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032. 2023-02-13 17:53:50,893 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032. after waiting 0 ms 2023-02-13 17:53:50,893 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032. 2023-02-13 17:53:50,893 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032. 2023-02-13 17:53:50,893 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1557): Region close journal for 52ceb95a47ba04015d52c77bc1874032: Waiting for close lock at 1676310830893Disabling compacts and flushes for region at 1676310830893Disabling writes for close at 1676310830893Writing region close event to WAL at 1676310830893Closed at 1676310830893 2023-02-13 17:53:50,896 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 17:53:50,897 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676310830897"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676310830897"}]},"ts":"1676310830897"} 2023-02-13 17:53:50,899 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 17:53:50,900 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 17:53:50,901 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableTwo","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676310830901"}]},"ts":"1676310830901"} 2023-02-13 17:53:50,902 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=restartTableTwo, state=ENABLING in hbase:meta 2023-02-13 17:53:50,906 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 17:53:50,907 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 17:53:50,907 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 17:53:50,907 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 17:53:50,907 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 17:53:50,908 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=52ceb95a47ba04015d52c77bc1874032, ASSIGN}] 2023-02-13 17:53:50,912 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=52ceb95a47ba04015d52c77bc1874032, ASSIGN 2023-02-13 17:53:50,914 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableTwo, region=52ceb95a47ba04015d52c77bc1874032, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42905,1676310828981; forceNewPlan=false, retain=false 2023-02-13 17:53:50,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46373] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 17:53:51,065 INFO [jenkins-hbase4:46373] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 17:53:51,066 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=52ceb95a47ba04015d52c77bc1874032, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42905,1676310828981 2023-02-13 17:53:51,066 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676310831066"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676310831066"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676310831066"}]},"ts":"1676310831066"} 2023-02-13 17:53:51,069 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; OpenRegionProcedure 52ceb95a47ba04015d52c77bc1874032, server=jenkins-hbase4.apache.org,42905,1676310828981}] 2023-02-13 17:53:51,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46373] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 17:53:51,225 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032. 2023-02-13 17:53:51,225 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 52ceb95a47ba04015d52c77bc1874032, NAME => 'restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032.', STARTKEY => '', ENDKEY => ''} 2023-02-13 17:53:51,225 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableTwo 52ceb95a47ba04015d52c77bc1874032 2023-02-13 17:53:51,225 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:51,225 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 52ceb95a47ba04015d52c77bc1874032 2023-02-13 17:53:51,225 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 52ceb95a47ba04015d52c77bc1874032 2023-02-13 17:53:51,227 INFO [StoreOpener-52ceb95a47ba04015d52c77bc1874032-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region 52ceb95a47ba04015d52c77bc1874032 2023-02-13 17:53:51,228 DEBUG [StoreOpener-52ceb95a47ba04015d52c77bc1874032-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableTwo/52ceb95a47ba04015d52c77bc1874032/family 2023-02-13 17:53:51,228 DEBUG [StoreOpener-52ceb95a47ba04015d52c77bc1874032-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableTwo/52ceb95a47ba04015d52c77bc1874032/family 2023-02-13 17:53:51,229 INFO [StoreOpener-52ceb95a47ba04015d52c77bc1874032-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 52ceb95a47ba04015d52c77bc1874032 columnFamilyName family 2023-02-13 17:53:51,229 INFO [StoreOpener-52ceb95a47ba04015d52c77bc1874032-1] regionserver.HStore(310): Store=52ceb95a47ba04015d52c77bc1874032/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:51,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableTwo/52ceb95a47ba04015d52c77bc1874032 2023-02-13 17:53:51,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableTwo/52ceb95a47ba04015d52c77bc1874032 2023-02-13 17:53:51,234 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 52ceb95a47ba04015d52c77bc1874032 2023-02-13 17:53:51,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableTwo/52ceb95a47ba04015d52c77bc1874032/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:53:51,236 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 52ceb95a47ba04015d52c77bc1874032; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65612846, jitterRate=-0.022292405366897583}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:53:51,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 52ceb95a47ba04015d52c77bc1874032: Running coprocessor pre-open hook at 1676310831226Writing region info on filesystem at 1676310831226Initializing all the Stores at 1676310831226Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676310831226Cleaning up temporary data from old regions at 1676310831231 (+5 ms)Cleaning up detritus from prior splits at 1676310831231Running coprocessor post-open hooks at 1676310831236 (+5 ms)Region opened successfully at 1676310831236 2023-02-13 17:53:51,237 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032., pid=14, masterSystemTime=1676310831221 2023-02-13 17:53:51,239 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032. 2023-02-13 17:53:51,239 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032. 2023-02-13 17:53:51,239 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=52ceb95a47ba04015d52c77bc1874032, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42905,1676310828981 2023-02-13 17:53:51,240 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676310831239"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676310831239"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676310831239"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676310831239"}]},"ts":"1676310831239"} 2023-02-13 17:53:51,244 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-02-13 17:53:51,244 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; OpenRegionProcedure 52ceb95a47ba04015d52c77bc1874032, server=jenkins-hbase4.apache.org,42905,1676310828981 in 173 msec 2023-02-13 17:53:51,246 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-02-13 17:53:51,246 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=restartTableTwo, region=52ceb95a47ba04015d52c77bc1874032, ASSIGN in 337 msec 2023-02-13 17:53:51,247 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 17:53:51,247 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableTwo","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676310831247"}]},"ts":"1676310831247"} 2023-02-13 17:53:51,248 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=restartTableTwo, state=ENABLED in hbase:meta 2023-02-13 17:53:51,253 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 17:53:51,254 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; CreateTableProcedure table=restartTableTwo in 409 msec 2023-02-13 17:53:51,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46373] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 17:53:51,454 INFO [Listener at localhost/40181] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableTwo, procId: 12 completed 2023-02-13 17:53:51,454 DEBUG [Listener at localhost/40181] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableTwo get assigned. Timeout = 60000ms 2023-02-13 17:53:51,454 INFO [Listener at localhost/40181] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:51,457 INFO [Listener at localhost/40181] hbase.HBaseTestingUtility(3484): All regions for table restartTableTwo assigned to meta. Checking AM states. 2023-02-13 17:53:51,457 INFO [Listener at localhost/40181] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:51,457 INFO [Listener at localhost/40181] hbase.HBaseTestingUtility(3504): All regions for table restartTableTwo assigned. 2023-02-13 17:53:51,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46373] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'restartTableThree', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 17:53:51,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46373] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableThree 2023-02-13 17:53:51,462 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 17:53:51,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46373] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "restartTableThree" procId is: 15 2023-02-13 17:53:51,463 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 17:53:51,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46373] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 17:53:51,465 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/.tmp/data/default/restartTableThree/d92fcd787d4162ea4435359fae28fa7d 2023-02-13 17:53:51,466 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/.tmp/data/default/restartTableThree/d92fcd787d4162ea4435359fae28fa7d empty. 2023-02-13 17:53:51,466 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/.tmp/data/default/restartTableThree/d92fcd787d4162ea4435359fae28fa7d 2023-02-13 17:53:51,466 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived restartTableThree regions 2023-02-13 17:53:51,473 WARN [IPC Server handler 2 on default port 36959] blockmanagement.BlockPlacementPolicyDefault(446): Failed to place enough replicas, still in need of 1 to reach 3 (unavailableStorages=[], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy and org.apache.hadoop.net.NetworkTopology 2023-02-13 17:53:51,473 WARN [IPC Server handler 2 on default port 36959] protocol.BlockStoragePolicy(161): Failed to place enough replicas: expected size is 1 but only 0 storage types can be selected (replication=3, selected=[], unavailable=[DISK], removed=[DISK], policy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}) 2023-02-13 17:53:51,473 WARN [IPC Server handler 2 on default port 36959] blockmanagement.BlockPlacementPolicyDefault(446): Failed to place enough replicas, still in need of 1 to reach 3 (unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) All required storage types are unavailable: unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]} 2023-02-13 17:53:51,481 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/.tmp/data/default/restartTableThree/.tabledesc/.tableinfo.0000000001 2023-02-13 17:53:51,483 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(7671): creating {ENCODED => d92fcd787d4162ea4435359fae28fa7d, NAME => 'restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableThree', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/.tmp 2023-02-13 17:53:51,486 WARN [IPC Server handler 2 on default port 36959] blockmanagement.BlockPlacementPolicyDefault(446): Failed to place enough replicas, still in need of 1 to reach 3 (unavailableStorages=[], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy and org.apache.hadoop.net.NetworkTopology 2023-02-13 17:53:51,486 WARN [IPC Server handler 2 on default port 36959] protocol.BlockStoragePolicy(161): Failed to place enough replicas: expected size is 1 but only 0 storage types can be selected (replication=3, selected=[], unavailable=[DISK], removed=[DISK], policy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}) 2023-02-13 17:53:51,486 WARN [IPC Server handler 2 on default port 36959] blockmanagement.BlockPlacementPolicyDefault(446): Failed to place enough replicas, still in need of 1 to reach 3 (unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) All required storage types are unavailable: unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]} 2023-02-13 17:53:51,492 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(865): Instantiated restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:51,493 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1603): Closing d92fcd787d4162ea4435359fae28fa7d, disabling compactions & flushes 2023-02-13 17:53:51,493 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1625): Closing region restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d. 2023-02-13 17:53:51,493 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d. 2023-02-13 17:53:51,493 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d. after waiting 0 ms 2023-02-13 17:53:51,493 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d. 2023-02-13 17:53:51,493 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1837): Closed restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d. 2023-02-13 17:53:51,493 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1557): Region close journal for d92fcd787d4162ea4435359fae28fa7d: Waiting for close lock at 1676310831493Disabling compacts and flushes for region at 1676310831493Disabling writes for close at 1676310831493Writing region close event to WAL at 1676310831493Closed at 1676310831493 2023-02-13 17:53:51,495 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 17:53:51,497 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676310831496"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676310831496"}]},"ts":"1676310831496"} 2023-02-13 17:53:51,499 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 17:53:51,501 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 17:53:51,501 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableThree","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676310831501"}]},"ts":"1676310831501"} 2023-02-13 17:53:51,503 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=restartTableThree, state=ENABLING in hbase:meta 2023-02-13 17:53:51,506 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 17:53:51,507 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 17:53:51,507 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 17:53:51,507 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 17:53:51,507 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 17:53:51,507 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=d92fcd787d4162ea4435359fae28fa7d, ASSIGN}] 2023-02-13 17:53:51,509 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=d92fcd787d4162ea4435359fae28fa7d, ASSIGN 2023-02-13 17:53:51,511 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableThree, region=d92fcd787d4162ea4435359fae28fa7d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34875,1676310828951; forceNewPlan=false, retain=false 2023-02-13 17:53:51,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46373] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 17:53:51,661 INFO [jenkins-hbase4:46373] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 17:53:51,662 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=d92fcd787d4162ea4435359fae28fa7d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34875,1676310828951 2023-02-13 17:53:51,663 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676310831662"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676310831662"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676310831662"}]},"ts":"1676310831662"} 2023-02-13 17:53:51,667 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE; OpenRegionProcedure d92fcd787d4162ea4435359fae28fa7d, server=jenkins-hbase4.apache.org,34875,1676310828951}] 2023-02-13 17:53:51,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46373] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 17:53:51,823 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d. 2023-02-13 17:53:51,823 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d92fcd787d4162ea4435359fae28fa7d, NAME => 'restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d.', STARTKEY => '', ENDKEY => ''} 2023-02-13 17:53:51,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableThree d92fcd787d4162ea4435359fae28fa7d 2023-02-13 17:53:51,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:51,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d92fcd787d4162ea4435359fae28fa7d 2023-02-13 17:53:51,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d92fcd787d4162ea4435359fae28fa7d 2023-02-13 17:53:51,825 INFO [StoreOpener-d92fcd787d4162ea4435359fae28fa7d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region d92fcd787d4162ea4435359fae28fa7d 2023-02-13 17:53:51,827 DEBUG [StoreOpener-d92fcd787d4162ea4435359fae28fa7d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableThree/d92fcd787d4162ea4435359fae28fa7d/family 2023-02-13 17:53:51,827 DEBUG [StoreOpener-d92fcd787d4162ea4435359fae28fa7d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableThree/d92fcd787d4162ea4435359fae28fa7d/family 2023-02-13 17:53:51,828 INFO [StoreOpener-d92fcd787d4162ea4435359fae28fa7d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d92fcd787d4162ea4435359fae28fa7d columnFamilyName family 2023-02-13 17:53:51,828 INFO [StoreOpener-d92fcd787d4162ea4435359fae28fa7d-1] regionserver.HStore(310): Store=d92fcd787d4162ea4435359fae28fa7d/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:51,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableThree/d92fcd787d4162ea4435359fae28fa7d 2023-02-13 17:53:51,830 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableThree/d92fcd787d4162ea4435359fae28fa7d 2023-02-13 17:53:51,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d92fcd787d4162ea4435359fae28fa7d 2023-02-13 17:53:51,834 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableThree/d92fcd787d4162ea4435359fae28fa7d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 17:53:51,835 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d92fcd787d4162ea4435359fae28fa7d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73302474, jitterRate=0.09229198098182678}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:53:51,835 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d92fcd787d4162ea4435359fae28fa7d: Running coprocessor pre-open hook at 1676310831824Writing region info on filesystem at 1676310831824Initializing all the Stores at 1676310831825 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676310831825Cleaning up temporary data from old regions at 1676310831830 (+5 ms)Cleaning up detritus from prior splits at 1676310831830Running coprocessor post-open hooks at 1676310831835 (+5 ms)Region opened successfully at 1676310831835 2023-02-13 17:53:51,836 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d., pid=17, masterSystemTime=1676310831819 2023-02-13 17:53:51,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d. 2023-02-13 17:53:51,837 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d. 2023-02-13 17:53:51,838 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=d92fcd787d4162ea4435359fae28fa7d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34875,1676310828951 2023-02-13 17:53:51,838 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676310831838"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676310831838"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676310831838"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676310831838"}]},"ts":"1676310831838"} 2023-02-13 17:53:51,842 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-02-13 17:53:51,842 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; OpenRegionProcedure d92fcd787d4162ea4435359fae28fa7d, server=jenkins-hbase4.apache.org,34875,1676310828951 in 173 msec 2023-02-13 17:53:51,845 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-02-13 17:53:51,845 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; TransitRegionStateProcedure table=restartTableThree, region=d92fcd787d4162ea4435359fae28fa7d, ASSIGN in 335 msec 2023-02-13 17:53:51,846 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 17:53:51,846 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableThree","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676310831846"}]},"ts":"1676310831846"} 2023-02-13 17:53:51,848 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=restartTableThree, state=ENABLED in hbase:meta 2023-02-13 17:53:51,850 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 17:53:51,852 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; CreateTableProcedure table=restartTableThree in 391 msec 2023-02-13 17:53:52,034 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-02-13 17:53:52,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46373] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 17:53:52,069 INFO [Listener at localhost/40181] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableThree, procId: 15 completed 2023-02-13 17:53:52,069 DEBUG [Listener at localhost/40181] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableThree get assigned. Timeout = 60000ms 2023-02-13 17:53:52,069 INFO [Listener at localhost/40181] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:52,072 INFO [Listener at localhost/40181] hbase.HBaseTestingUtility(3484): All regions for table restartTableThree assigned to meta. Checking AM states. 2023-02-13 17:53:52,072 INFO [Listener at localhost/40181] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:52,072 INFO [Listener at localhost/40181] hbase.HBaseTestingUtility(3504): All regions for table restartTableThree assigned. 2023-02-13 17:53:52,073 INFO [Listener at localhost/40181] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:52,081 INFO [Listener at localhost/40181] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:52,085 INFO [Listener at localhost/40181] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:52,088 INFO [Listener at localhost/40181] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:52,088 INFO [Listener at localhost/40181] master.SnapshotOfRegionAssignmentFromMeta(102): Start to scan the hbase:meta for the current region assignment snappshot 2023-02-13 17:53:52,092 INFO [Listener at localhost/40181] master.SnapshotOfRegionAssignmentFromMeta(164): Finished to scan the hbase:meta for the current region assignmentsnapshot 2023-02-13 17:53:52,092 INFO [Listener at localhost/40181] master.TestRetainAssignmentOnRestart(88): Shutting down HBase cluster 2023-02-13 17:53:52,093 INFO [Listener at localhost/40181] hbase.MiniHBaseCluster(631): Stopping Thread[M:0;jenkins-hbase4:46373,5,FailOnTimeoutGroup] 2023-02-13 17:53:52,093 INFO [Listener at localhost/40181] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,46373,1676310828881' ***** 2023-02-13 17:53:52,093 INFO [Listener at localhost/40181] regionserver.HRegionServer(2310): STOPPED: Stopping master 0 2023-02-13 17:53:52,094 DEBUG [M:0;jenkins-hbase4:46373] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59e60571, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 17:53:52,094 INFO [M:0;jenkins-hbase4:46373] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,46373,1676310828881 2023-02-13 17:53:52,094 DEBUG [M:0;jenkins-hbase4:46373] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28d37515 to 127.0.0.1:53630 2023-02-13 17:53:52,095 DEBUG [M:0;jenkins-hbase4:46373] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:52,095 INFO [M:0;jenkins-hbase4:46373] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,46373,1676310828881; all regions closed. 2023-02-13 17:53:52,095 DEBUG [M:0;jenkins-hbase4:46373] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:52,095 DEBUG [M:0;jenkins-hbase4:46373] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-13 17:53:52,095 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-13 17:53:52,096 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676310829211] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676310829211,5,FailOnTimeoutGroup] 2023-02-13 17:53:52,096 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676310829215] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676310829215,5,FailOnTimeoutGroup] 2023-02-13 17:53:52,095 DEBUG [M:0;jenkins-hbase4:46373] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-13 17:53:52,098 INFO [M:0;jenkins-hbase4:46373] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-13 17:53:52,098 INFO [M:0;jenkins-hbase4:46373] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-13 17:53:52,098 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:39461-0x1004ed140fc0001, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 17:53:52,098 INFO [M:0;jenkins-hbase4:46373] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-13 17:53:52,098 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:34875-0x1004ed140fc0002, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 17:53:52,098 DEBUG [M:0;jenkins-hbase4:46373] master.HMaster(1502): Stopping service threads 2023-02-13 17:53:52,098 INFO [M:0;jenkins-hbase4:46373] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-13 17:53:52,098 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:42905-0x1004ed140fc0003, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 17:53:52,098 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 17:53:52,099 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:53:52,098 DEBUG [Listener at localhost/40181] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-13 17:53:52,099 INFO [M:0;jenkins-hbase4:46373] procedure2.ProcedureExecutor(629): Stopping 2023-02-13 17:53:52,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34875-0x1004ed140fc0002, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 17:53:52,099 INFO [Listener at localhost/40181] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,39461,1676310828923' ***** 2023-02-13 17:53:52,100 INFO [Listener at localhost/40181] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 17:53:52,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39461-0x1004ed140fc0001, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 17:53:52,101 INFO [RS:0;jenkins-hbase4:39461] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 17:53:52,101 ERROR [M:0;jenkins-hbase4:46373] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-5,5,PEWorkerGroup] Thread[HFileArchiver-6,5,PEWorkerGroup] Thread[HFileArchiver-7,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-02-13 17:53:52,101 INFO [Listener at localhost/40181] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,34875,1676310828951' ***** 2023-02-13 17:53:52,101 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 17:53:52,101 INFO [M:0;jenkins-hbase4:46373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-13 17:53:52,101 INFO [RS:0;jenkins-hbase4:39461] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 17:53:52,102 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-13 17:53:52,102 INFO [RS:0;jenkins-hbase4:39461] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 17:53:52,101 INFO [Listener at localhost/40181] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 17:53:52,102 INFO [RS:0;jenkins-hbase4:39461] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,39461,1676310828923 2023-02-13 17:53:52,102 INFO [Listener at localhost/40181] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,42905,1676310828981' ***** 2023-02-13 17:53:52,102 INFO [RS:1;jenkins-hbase4:34875] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 17:53:52,102 INFO [Listener at localhost/40181] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 17:53:52,102 INFO [RS:1;jenkins-hbase4:34875] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 17:53:52,102 INFO [RS:1;jenkins-hbase4:34875] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 17:53:52,102 INFO [RS:2;jenkins-hbase4:42905] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 17:53:52,102 DEBUG [RS:0;jenkins-hbase4:39461] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x741cd5b4 to 127.0.0.1:53630 2023-02-13 17:53:52,103 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 17:53:52,103 INFO [RS:2;jenkins-hbase4:42905] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 17:53:52,102 INFO [RS:1;jenkins-hbase4:34875] regionserver.HRegionServer(3304): Received CLOSE for d92fcd787d4162ea4435359fae28fa7d 2023-02-13 17:53:52,102 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 17:53:52,103 INFO [RS:2;jenkins-hbase4:42905] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 17:53:52,103 DEBUG [RS:0;jenkins-hbase4:39461] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:52,104 INFO [RS:1;jenkins-hbase4:34875] regionserver.HRegionServer(3304): Received CLOSE for 28d66b4066b494c8b8644327cd7fdb67 2023-02-13 17:53:52,104 INFO [RS:2;jenkins-hbase4:42905] regionserver.HRegionServer(3304): Received CLOSE for 52ceb95a47ba04015d52c77bc1874032 2023-02-13 17:53:52,105 INFO [RS:1;jenkins-hbase4:34875] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,34875,1676310828951 2023-02-13 17:53:52,105 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d92fcd787d4162ea4435359fae28fa7d, disabling compactions & flushes 2023-02-13 17:53:52,105 DEBUG [RS:1;jenkins-hbase4:34875] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x72b79309 to 127.0.0.1:53630 2023-02-13 17:53:52,105 DEBUG [RS:1;jenkins-hbase4:34875] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:52,105 INFO [RS:1;jenkins-hbase4:34875] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-13 17:53:52,105 DEBUG [RS:1;jenkins-hbase4:34875] regionserver.HRegionServer(1479): Online Regions={d92fcd787d4162ea4435359fae28fa7d=restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d., 28d66b4066b494c8b8644327cd7fdb67=hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67.} 2023-02-13 17:53:52,105 DEBUG [RS:1;jenkins-hbase4:34875] regionserver.HRegionServer(1505): Waiting on 28d66b4066b494c8b8644327cd7fdb67, d92fcd787d4162ea4435359fae28fa7d 2023-02-13 17:53:52,104 INFO [RS:0;jenkins-hbase4:39461] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,39461,1676310828923; all regions closed. 2023-02-13 17:53:52,105 INFO [RS:2;jenkins-hbase4:42905] regionserver.HRegionServer(3304): Received CLOSE for fbc323e3b53470e4b5a74c9688484c26 2023-02-13 17:53:52,105 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d. 2023-02-13 17:53:52,105 INFO [RS:2;jenkins-hbase4:42905] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,42905,1676310828981 2023-02-13 17:53:52,105 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d. 2023-02-13 17:53:52,105 DEBUG [RS:2;jenkins-hbase4:42905] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20c49cc5 to 127.0.0.1:53630 2023-02-13 17:53:52,105 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d. after waiting 0 ms 2023-02-13 17:53:52,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d. 2023-02-13 17:53:52,106 DEBUG [RS:2;jenkins-hbase4:42905] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:52,106 INFO [RS:2;jenkins-hbase4:42905] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 17:53:52,106 INFO [RS:2;jenkins-hbase4:42905] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 17:53:52,106 INFO [RS:2;jenkins-hbase4:42905] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 17:53:52,106 INFO [RS:2;jenkins-hbase4:42905] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-13 17:53:52,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 52ceb95a47ba04015d52c77bc1874032, disabling compactions & flushes 2023-02-13 17:53:52,107 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032. 2023-02-13 17:53:52,107 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032. 2023-02-13 17:53:52,107 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032. after waiting 0 ms 2023-02-13 17:53:52,107 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032. 2023-02-13 17:53:52,107 INFO [RS:2;jenkins-hbase4:42905] regionserver.HRegionServer(1475): Waiting on 3 regions to close 2023-02-13 17:53:52,107 DEBUG [RS:2;jenkins-hbase4:42905] regionserver.HRegionServer(1479): Online Regions={52ceb95a47ba04015d52c77bc1874032=restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032., fbc323e3b53470e4b5a74c9688484c26=restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26., 1588230740=hbase:meta,,1.1588230740} 2023-02-13 17:53:52,107 DEBUG [RS:2;jenkins-hbase4:42905] regionserver.HRegionServer(1505): Waiting on 1588230740, 52ceb95a47ba04015d52c77bc1874032, fbc323e3b53470e4b5a74c9688484c26 2023-02-13 17:53:52,107 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 17:53:52,107 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 17:53:52,107 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 17:53:52,107 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 17:53:52,107 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 17:53:52,108 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=5.14 KB heapSize=9.42 KB 2023-02-13 17:53:52,120 DEBUG [RS:0;jenkins-hbase4:39461] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/oldWALs 2023-02-13 17:53:52,120 INFO [RS:0;jenkins-hbase4:39461] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C39461%2C1676310828923:(num 1676310829464) 2023-02-13 17:53:52,120 DEBUG [RS:0;jenkins-hbase4:39461] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:52,121 INFO [RS:0;jenkins-hbase4:39461] regionserver.LeaseManager(133): Closed leases 2023-02-13 17:53:52,122 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 17:53:52,122 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableThree/d92fcd787d4162ea4435359fae28fa7d/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 17:53:52,126 INFO [RS:0;jenkins-hbase4:39461] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-13 17:53:52,126 INFO [RS:0;jenkins-hbase4:39461] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 17:53:52,126 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 17:53:52,126 INFO [RS:0;jenkins-hbase4:39461] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 17:53:52,126 INFO [RS:0;jenkins-hbase4:39461] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 17:53:52,127 INFO [RS:0;jenkins-hbase4:39461] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39461 2023-02-13 17:53:52,133 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 17:53:52,134 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d. 2023-02-13 17:53:52,134 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d92fcd787d4162ea4435359fae28fa7d: Waiting for close lock at 1676310832104Running coprocessor pre-close hooks at 1676310832104Disabling compacts and flushes for region at 1676310832104Disabling writes for close at 1676310832106 (+2 ms)Writing region close event to WAL at 1676310832107 (+1 ms)Running coprocessor post-close hooks at 1676310832127 (+20 ms)Closed at 1676310832134 (+7 ms) 2023-02-13 17:53:52,134 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d. 2023-02-13 17:53:52,140 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 28d66b4066b494c8b8644327cd7fdb67, disabling compactions & flushes 2023-02-13 17:53:52,140 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 17:53:52,140 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67. 2023-02-13 17:53:52,141 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67. 2023-02-13 17:53:52,141 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67. after waiting 0 ms 2023-02-13 17:53:52,141 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67. 2023-02-13 17:53:52,141 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 28d66b4066b494c8b8644327cd7fdb67 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-13 17:53:52,142 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableTwo/52ceb95a47ba04015d52c77bc1874032/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 17:53:52,145 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032. 2023-02-13 17:53:52,145 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 52ceb95a47ba04015d52c77bc1874032: Waiting for close lock at 1676310832105Running coprocessor pre-close hooks at 1676310832105Disabling compacts and flushes for region at 1676310832105Disabling writes for close at 1676310832107 (+2 ms)Writing region close event to WAL at 1676310832108 (+1 ms)Running coprocessor post-close hooks at 1676310832143 (+35 ms)Closed at 1676310832145 (+2 ms) 2023-02-13 17:53:52,145 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032. 2023-02-13 17:53:52,146 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing fbc323e3b53470e4b5a74c9688484c26, disabling compactions & flushes 2023-02-13 17:53:52,146 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26. 2023-02-13 17:53:52,146 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26. 2023-02-13 17:53:52,146 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26. after waiting 0 ms 2023-02-13 17:53:52,146 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26. 2023-02-13 17:53:52,168 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableOne/fbc323e3b53470e4b5a74c9688484c26/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 17:53:52,171 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26. 2023-02-13 17:53:52,171 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for fbc323e3b53470e4b5a74c9688484c26: Waiting for close lock at 1676310832145Running coprocessor pre-close hooks at 1676310832145Disabling compacts and flushes for region at 1676310832145Disabling writes for close at 1676310832146 (+1 ms)Writing region close event to WAL at 1676310832146Running coprocessor post-close hooks at 1676310832169 (+23 ms)Closed at 1676310832171 (+2 ms) 2023-02-13 17:53:52,171 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26. 2023-02-13 17:53:52,189 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.77 KB at sequenceid=24 (bloomFilter=false), to=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/.tmp/info/0174fadab1e947fa930999dff816966f 2023-02-13 17:53:52,195 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/namespace/28d66b4066b494c8b8644327cd7fdb67/.tmp/info/438f5ae286fa473e94b8c90935fd5fb1 2023-02-13 17:53:52,201 DEBUG [M:0;jenkins-hbase4:46373] zookeeper.ZKUtil(398): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-13 17:53:52,201 WARN [M:0;jenkins-hbase4:46373] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-13 17:53:52,201 INFO [M:0;jenkins-hbase4:46373] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-13 17:53:52,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 17:53:52,205 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:34875-0x1004ed140fc0002, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39461,1676310828923 2023-02-13 17:53:52,205 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:42905-0x1004ed140fc0003, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39461,1676310828923 2023-02-13 17:53:52,205 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:39461-0x1004ed140fc0001, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39461,1676310828923 2023-02-13 17:53:52,205 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:42905-0x1004ed140fc0003, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:52,205 ERROR [Listener at localhost/40181-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6b9a8aaa rejected from java.util.concurrent.ThreadPoolExecutor@6fa51301[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:52,205 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:34875-0x1004ed140fc0002, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:52,205 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:52,205 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:39461-0x1004ed140fc0001, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:52,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42905-0x1004ed140fc0003, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 17:53:52,206 ERROR [Listener at localhost/40181-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@24b7fb44 rejected from java.util.concurrent.ThreadPoolExecutor@6fa51301[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:52,206 INFO [M:0;jenkins-hbase4:46373] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-13 17:53:52,207 DEBUG [M:0;jenkins-hbase4:46373] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 17:53:52,207 INFO [M:0;jenkins-hbase4:46373] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:53:52,207 DEBUG [M:0;jenkins-hbase4:46373] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:53:52,207 DEBUG [M:0;jenkins-hbase4:46373] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 17:53:52,207 DEBUG [M:0;jenkins-hbase4:46373] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:53:52,207 INFO [M:0;jenkins-hbase4:46373] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=64.23 KB heapSize=76.14 KB 2023-02-13 17:53:52,211 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/namespace/28d66b4066b494c8b8644327cd7fdb67/.tmp/info/438f5ae286fa473e94b8c90935fd5fb1 as hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/namespace/28d66b4066b494c8b8644327cd7fdb67/info/438f5ae286fa473e94b8c90935fd5fb1 2023-02-13 17:53:52,222 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/namespace/28d66b4066b494c8b8644327cd7fdb67/info/438f5ae286fa473e94b8c90935fd5fb1, entries=2, sequenceid=6, filesize=4.8 K 2023-02-13 17:53:52,223 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 28d66b4066b494c8b8644327cd7fdb67 in 82ms, sequenceid=6, compaction requested=false 2023-02-13 17:53:52,224 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-13 17:53:52,224 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=380 B at sequenceid=24 (bloomFilter=false), to=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/.tmp/table/c5d2fb40a5f947c88b9739f47ec9436c 2023-02-13 17:53:52,232 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/.tmp/info/0174fadab1e947fa930999dff816966f as hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/info/0174fadab1e947fa930999dff816966f 2023-02-13 17:53:52,239 INFO [M:0;jenkins-hbase4:46373] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.23 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/5e57838929254d419cea807721036b05 2023-02-13 17:53:52,241 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/namespace/28d66b4066b494c8b8644327cd7fdb67/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-13 17:53:52,241 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/info/0174fadab1e947fa930999dff816966f, entries=40, sequenceid=24, filesize=9.5 K 2023-02-13 17:53:52,242 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67. 2023-02-13 17:53:52,242 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 28d66b4066b494c8b8644327cd7fdb67: Waiting for close lock at 1676310832134Running coprocessor pre-close hooks at 1676310832134Disabling compacts and flushes for region at 1676310832134Disabling writes for close at 1676310832141 (+7 ms)Obtaining lock to block concurrent updates at 1676310832141Preparing flush snapshotting stores in 28d66b4066b494c8b8644327cd7fdb67 at 1676310832141Finished memstore snapshotting hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1676310832141Flushing stores of hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67. at 1676310832142 (+1 ms)Flushing 28d66b4066b494c8b8644327cd7fdb67/info: creating writer at 1676310832142Flushing 28d66b4066b494c8b8644327cd7fdb67/info: appending metadata at 1676310832169 (+27 ms)Flushing 28d66b4066b494c8b8644327cd7fdb67/info: closing flushed file at 1676310832169Flushing 28d66b4066b494c8b8644327cd7fdb67/info: reopening flushed file at 1676310832212 (+43 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 28d66b4066b494c8b8644327cd7fdb67 in 82ms, sequenceid=6, compaction requested=false at 1676310832224 (+12 ms)Writing region close event to WAL at 1676310832234 (+10 ms)Running coprocessor post-close hooks at 1676310832242 (+8 ms)Closed at 1676310832242 2023-02-13 17:53:52,243 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67. 2023-02-13 17:53:52,243 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/.tmp/table/c5d2fb40a5f947c88b9739f47ec9436c as hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/table/c5d2fb40a5f947c88b9739f47ec9436c 2023-02-13 17:53:52,246 DEBUG [M:0;jenkins-hbase4:46373] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/5e57838929254d419cea807721036b05 as hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/5e57838929254d419cea807721036b05 2023-02-13 17:53:52,249 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/table/c5d2fb40a5f947c88b9739f47ec9436c, entries=8, sequenceid=24, filesize=5.0 K 2023-02-13 17:53:52,250 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~5.14 KB/5262, heapSize ~9.14 KB/9360, currentSize=0 B/0 for 1588230740 in 142ms, sequenceid=24, compaction requested=false 2023-02-13 17:53:52,250 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-13 17:53:52,255 INFO [M:0;jenkins-hbase4:46373] regionserver.HStore(1080): Added hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/5e57838929254d419cea807721036b05, entries=17, sequenceid=141, filesize=8.5 K 2023-02-13 17:53:52,256 INFO [M:0;jenkins-hbase4:46373] regionserver.HRegion(2947): Finished flush of dataSize ~64.23 KB/65767, heapSize ~76.13 KB/77952, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 49ms, sequenceid=141, compaction requested=false 2023-02-13 17:53:52,258 INFO [M:0;jenkins-hbase4:46373] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:53:52,258 DEBUG [M:0;jenkins-hbase4:46373] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676310832206Disabling compacts and flushes for region at 1676310832206Disabling writes for close at 1676310832207 (+1 ms)Obtaining lock to block concurrent updates at 1676310832207Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676310832207Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=65767, getHeapSize=77952, getOffHeapSize=0, getCellsCount=158 at 1676310832208 (+1 ms)Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676310832209 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676310832209Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676310832213 (+4 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676310832214 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676310832247 (+33 ms)Finished flush of dataSize ~64.23 KB/65767, heapSize ~76.13 KB/77952, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 49ms, sequenceid=141, compaction requested=false at 1676310832256 (+9 ms)Writing region close event to WAL at 1676310832258 (+2 ms)Closed at 1676310832258 2023-02-13 17:53:52,261 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=1 2023-02-13 17:53:52,262 INFO [M:0;jenkins-hbase4:46373] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-13 17:53:52,262 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 17:53:52,262 INFO [M:0;jenkins-hbase4:46373] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46373 2023-02-13 17:53:52,262 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 17:53:52,263 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 17:53:52,263 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676310832107Running coprocessor pre-close hooks at 1676310832107Disabling compacts and flushes for region at 1676310832107Disabling writes for close at 1676310832107Obtaining lock to block concurrent updates at 1676310832108 (+1 ms)Preparing flush snapshotting stores in 1588230740 at 1676310832108Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=5262, getHeapSize=9600, getOffHeapSize=0, getCellsCount=48 at 1676310832109 (+1 ms)Flushing stores of hbase:meta,,1.1588230740 at 1676310832110 (+1 ms)Flushing 1588230740/info: creating writer at 1676310832110Flushing 1588230740/info: appending metadata at 1676310832143 (+33 ms)Flushing 1588230740/info: closing flushed file at 1676310832143Flushing 1588230740/table: creating writer at 1676310832197 (+54 ms)Flushing 1588230740/table: appending metadata at 1676310832207 (+10 ms)Flushing 1588230740/table: closing flushed file at 1676310832207Flushing 1588230740/info: reopening flushed file at 1676310832233 (+26 ms)Flushing 1588230740/table: reopening flushed file at 1676310832243 (+10 ms)Finished flush of dataSize ~5.14 KB/5262, heapSize ~9.14 KB/9360, currentSize=0 B/0 for 1588230740 in 142ms, sequenceid=24, compaction requested=false at 1676310832250 (+7 ms)Writing region close event to WAL at 1676310832255 (+5 ms)Running coprocessor post-close hooks at 1676310832262 (+7 ms)Closed at 1676310832263 (+1 ms) 2023-02-13 17:53:52,264 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-13 17:53:52,306 INFO [RS:1;jenkins-hbase4:34875] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,34875,1676310828951; all regions closed. 2023-02-13 17:53:52,307 INFO [RS:2;jenkins-hbase4:42905] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,42905,1676310828981; all regions closed. 2023-02-13 17:53:52,310 DEBUG [M:0;jenkins-hbase4:46373] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,46373,1676310828881 already deleted, retry=false 2023-02-13 17:53:52,316 DEBUG [RS:1;jenkins-hbase4:34875] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/oldWALs 2023-02-13 17:53:52,316 INFO [RS:1;jenkins-hbase4:34875] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34875%2C1676310828951:(num 1676310829459) 2023-02-13 17:53:52,316 DEBUG [RS:1;jenkins-hbase4:34875] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:52,316 INFO [RS:1;jenkins-hbase4:34875] regionserver.LeaseManager(133): Closed leases 2023-02-13 17:53:52,316 DEBUG [RS:2;jenkins-hbase4:42905] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/oldWALs 2023-02-13 17:53:52,316 INFO [RS:2;jenkins-hbase4:42905] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C42905%2C1676310828981.meta:.meta(num 1676310829672) 2023-02-13 17:53:52,316 INFO [RS:1;jenkins-hbase4:34875] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 17:53:52,316 INFO [RS:1;jenkins-hbase4:34875] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 17:53:52,316 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 17:53:52,316 INFO [RS:1;jenkins-hbase4:34875] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 17:53:52,316 INFO [RS:1;jenkins-hbase4:34875] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 17:53:52,317 INFO [RS:1;jenkins-hbase4:34875] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34875 2023-02-13 17:53:52,321 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:42905-0x1004ed140fc0003, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34875,1676310828951 2023-02-13 17:53:52,321 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:34875-0x1004ed140fc0002, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34875,1676310828951 2023-02-13 17:53:52,327 DEBUG [RS:2;jenkins-hbase4:42905] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/oldWALs 2023-02-13 17:53:52,327 INFO [RS:2;jenkins-hbase4:42905] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C42905%2C1676310828981:(num 1676310829467) 2023-02-13 17:53:52,327 DEBUG [RS:2;jenkins-hbase4:42905] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:52,327 INFO [RS:2;jenkins-hbase4:42905] regionserver.LeaseManager(133): Closed leases 2023-02-13 17:53:52,327 INFO [RS:2;jenkins-hbase4:42905] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 17:53:52,327 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 17:53:52,328 INFO [RS:2;jenkins-hbase4:42905] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42905 2023-02-13 17:53:52,329 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:42905-0x1004ed140fc0003, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42905,1676310828981 2023-02-13 17:53:52,329 ERROR [Listener at localhost/40181-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@57c4874a rejected from java.util.concurrent.ThreadPoolExecutor@53fb196a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:52,502 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:42905-0x1004ed140fc0003, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:52,502 INFO [RS:2;jenkins-hbase4:42905] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,42905,1676310828981; zookeeper connection closed. 2023-02-13 17:53:52,502 ERROR [Listener at localhost/40181-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5428ae16 rejected from java.util.concurrent.ThreadPoolExecutor@53fb196a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:52,504 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6a5bc6d2] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6a5bc6d2 2023-02-13 17:53:52,504 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:42905-0x1004ed140fc0003, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:52,504 ERROR [Listener at localhost/40181-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2c1ee0c9 rejected from java.util.concurrent.ThreadPoolExecutor@53fb196a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:52,602 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:34875-0x1004ed140fc0002, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:52,602 INFO [RS:1;jenkins-hbase4:34875] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,34875,1676310828951; zookeeper connection closed. 2023-02-13 17:53:52,602 ERROR [Listener at localhost/40181-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@70b7ddeb rejected from java.util.concurrent.ThreadPoolExecutor@74884725[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:52,603 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@74c93363] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@74c93363 2023-02-13 17:53:52,603 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:34875-0x1004ed140fc0002, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:52,603 ERROR [Listener at localhost/40181-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4e675730 rejected from java.util.concurrent.ThreadPoolExecutor@74884725[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:52,703 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:52,703 INFO [M:0;jenkins-hbase4:46373] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,46373,1676310828881; zookeeper connection closed. 2023-02-13 17:53:52,703 ERROR [Listener at localhost/40181-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@703d9fc9 rejected from java.util.concurrent.ThreadPoolExecutor@74020516[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 26] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:52,704 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:46373-0x1004ed140fc0000, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:52,704 ERROR [Listener at localhost/40181-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1c2454ea rejected from java.util.concurrent.ThreadPoolExecutor@74020516[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 26] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:52,803 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:39461-0x1004ed140fc0001, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:52,803 INFO [RS:0;jenkins-hbase4:39461] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,39461,1676310828923; zookeeper connection closed. 2023-02-13 17:53:52,803 ERROR [Listener at localhost/40181-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5cbdcfd0 rejected from java.util.concurrent.ThreadPoolExecutor@6fa51301[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:52,803 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@54945514] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@54945514 2023-02-13 17:53:52,803 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:39461-0x1004ed140fc0001, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:52,803 ERROR [Listener at localhost/40181-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@517ec010 rejected from java.util.concurrent.ThreadPoolExecutor@6fa51301[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:52,803 INFO [Listener at localhost/40181] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-13 17:53:52,804 INFO [Listener at localhost/40181] master.TestRetainAssignmentOnRestart(93): Sleeping a bit 2023-02-13 17:53:54,804 INFO [Listener at localhost/40181] master.TestRetainAssignmentOnRestart(96): Starting cluster the second time with the same ports 2023-02-13 17:53:54,816 INFO [Listener at localhost/40181] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 17:53:54,816 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:54,817 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:54,817 INFO [Listener at localhost/40181] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 17:53:54,817 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:54,817 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 17:53:54,817 INFO [Listener at localhost/40181] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 17:53:54,818 INFO [Listener at localhost/40181] ipc.NettyRpcServer(120): Bind to /172.31.14.131:37835 2023-02-13 17:53:54,819 INFO [Listener at localhost/40181] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:54,820 INFO [Listener at localhost/40181] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:54,821 INFO [Listener at localhost/40181] zookeeper.RecoverableZooKeeper(93): Process identifier=master:37835 connecting to ZooKeeper ensemble=127.0.0.1:53630 2023-02-13 17:53:54,825 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:378350x0, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 17:53:54,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:37835-0x1004ed140fc0009 connected 2023-02-13 17:53:54,828 DEBUG [Listener at localhost/40181] zookeeper.ZKUtil(164): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 17:53:54,828 DEBUG [Listener at localhost/40181] zookeeper.ZKUtil(162): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-13 17:53:54,829 DEBUG [Listener at localhost/40181] zookeeper.ZKUtil(164): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 17:53:54,832 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37835 2023-02-13 17:53:54,832 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37835 2023-02-13 17:53:54,833 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37835 2023-02-13 17:53:54,833 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37835 2023-02-13 17:53:54,834 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37835 2023-02-13 17:53:54,834 INFO [Listener at localhost/40181] master.HMaster(439): hbase.rootdir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e, hbase.cluster.distributed=false 2023-02-13 17:53:54,835 DEBUG [pool-503-thread-1] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: INIT 2023-02-13 17:53:54,836 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,37835,1676310834816 2023-02-13 17:53:54,837 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 17:53:54,837 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,37835,1676310834816 2023-02-13 17:53:54,840 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 17:53:54,840 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:53:54,840 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 17:53:54,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 17:53:54,841 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,37835,1676310834816 from backup master directory 2023-02-13 17:53:54,841 INFO [M:1;jenkins-hbase4:37835] regionserver.HRegionServer(952): ClusterId : 2a4478ba-ba99-441c-8812-a0c5f19f8b8a 2023-02-13 17:53:54,842 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,37835,1676310834816 2023-02-13 17:53:54,842 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 17:53:54,842 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 17:53:54,842 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,37835,1676310834816 2023-02-13 17:53:54,849 INFO [Listener at localhost/40181] client.ConnectionUtils(127): regionserver/jenkins-hbase4:39461 server-side Connection retries=45 2023-02-13 17:53:54,850 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:54,850 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:54,850 INFO [Listener at localhost/40181] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 17:53:54,850 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:54,850 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 17:53:54,850 INFO [Listener at localhost/40181] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 17:53:54,852 INFO [Listener at localhost/40181] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39461 2023-02-13 17:53:54,852 INFO [Listener at localhost/40181] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 17:53:54,856 DEBUG [Listener at localhost/40181] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 17:53:54,856 INFO [Listener at localhost/40181] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:54,858 INFO [Listener at localhost/40181] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:54,859 INFO [Listener at localhost/40181] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39461 connecting to ZooKeeper ensemble=127.0.0.1:53630 2023-02-13 17:53:54,861 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:54,863 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:394610x0, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 17:53:54,864 DEBUG [Listener at localhost/40181] zookeeper.ZKUtil(162): regionserver:394610x0, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 17:53:54,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:39461-0x1004ed140fc000a connected 2023-02-13 17:53:54,865 DEBUG [Listener at localhost/40181] zookeeper.ZKUtil(162): regionserver:39461-0x1004ed140fc000a, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-13 17:53:54,866 DEBUG [Listener at localhost/40181] zookeeper.ZKUtil(164): regionserver:39461-0x1004ed140fc000a, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 17:53:54,867 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39461 2023-02-13 17:53:54,867 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39461 2023-02-13 17:53:54,867 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39461 2023-02-13 17:53:54,867 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39461 2023-02-13 17:53:54,868 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39461 2023-02-13 17:53:54,869 INFO [RS:3;jenkins-hbase4:39461] regionserver.HRegionServer(952): ClusterId : 2a4478ba-ba99-441c-8812-a0c5f19f8b8a 2023-02-13 17:53:54,869 DEBUG [RS:3;jenkins-hbase4:39461] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 17:53:54,871 DEBUG [RS:3;jenkins-hbase4:39461] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 17:53:54,871 DEBUG [RS:3;jenkins-hbase4:39461] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 17:53:54,879 DEBUG [RS:3;jenkins-hbase4:39461] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 17:53:54,880 DEBUG [RS:3;jenkins-hbase4:39461] zookeeper.ReadOnlyZKClient(139): Connect 0x0d7c72b1 to 127.0.0.1:53630 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:53:54,888 DEBUG [RS:3;jenkins-hbase4:39461] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3bb9d6e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:53:54,889 DEBUG [RS:3;jenkins-hbase4:39461] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78419c1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 17:53:54,900 DEBUG [RS:3;jenkins-hbase4:39461] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:3;jenkins-hbase4:39461 2023-02-13 17:53:54,900 INFO [RS:3;jenkins-hbase4:39461] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 17:53:54,900 INFO [RS:3;jenkins-hbase4:39461] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 17:53:54,900 DEBUG [RS:3;jenkins-hbase4:39461] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 17:53:54,901 INFO [RS:3;jenkins-hbase4:39461] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,37835,1676310834816 with isa=jenkins-hbase4.apache.org/172.31.14.131:39461, startcode=1676310834849 2023-02-13 17:53:54,902 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x77983962 to 127.0.0.1:53630 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:53:54,902 DEBUG [RS:3;jenkins-hbase4:39461] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 17:53:54,907 INFO [RS-EventLoopGroup-13-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54957, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.9 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 17:53:54,907 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ab76ca2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:53:54,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37835] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:53:54,908 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 17:53:54,908 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-13 17:53:54,908 DEBUG [RS:3;jenkins-hbase4:39461] regionserver.HRegionServer(2829): Master is not running yet 2023-02-13 17:53:54,909 WARN [RS:3;jenkins-hbase4:39461] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-13 17:53:54,909 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 17:53:54,915 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/WALs/jenkins-hbase4.apache.org,46373,1676310828881 to hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/WALs/jenkins-hbase4.apache.org,46373,1676310828881-dead as it is dead 2023-02-13 17:53:54,916 INFO [master/jenkins-hbase4:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/WALs/jenkins-hbase4.apache.org,46373,1676310828881-dead/jenkins-hbase4.apache.org%2C46373%2C1676310828881.1676310829113 2023-02-13 17:53:54,916 INFO [master/jenkins-hbase4:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/WALs/jenkins-hbase4.apache.org,46373,1676310828881-dead/jenkins-hbase4.apache.org%2C46373%2C1676310828881.1676310829113 after 0ms 2023-02-13 17:53:54,917 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/WALs/jenkins-hbase4.apache.org,46373,1676310828881-dead/jenkins-hbase4.apache.org%2C46373%2C1676310828881.1676310829113 to hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C46373%2C1676310828881.1676310829113 2023-02-13 17:53:54,917 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/WALs/jenkins-hbase4.apache.org,46373,1676310828881-dead 2023-02-13 17:53:54,918 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/WALs/jenkins-hbase4.apache.org,37835,1676310834816 2023-02-13 17:53:54,920 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C37835%2C1676310834816, suffix=, logDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/WALs/jenkins-hbase4.apache.org,37835,1676310834816, archiveDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/oldWALs, maxLogs=10 2023-02-13 17:53:54,936 DEBUG [RS-EventLoopGroup-14-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39523,DS-cf1e0e36-9598-4b66-ab37-608d409fcf08,DISK] 2023-02-13 17:53:54,936 DEBUG [RS-EventLoopGroup-14-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37061,DS-c817cfbe-f58e-4f4d-94e0-14281a32ca4c,DISK] 2023-02-13 17:53:54,936 DEBUG [RS-EventLoopGroup-14-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37727,DS-c79d1410-43a1-4b12-a2c2-dff1472abceb,DISK] 2023-02-13 17:53:54,940 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/WALs/jenkins-hbase4.apache.org,37835,1676310834816/jenkins-hbase4.apache.org%2C37835%2C1676310834816.1676310834920 2023-02-13 17:53:54,943 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39523,DS-cf1e0e36-9598-4b66-ab37-608d409fcf08,DISK], DatanodeInfoWithStorage[127.0.0.1:37727,DS-c79d1410-43a1-4b12-a2c2-dff1472abceb,DISK], DatanodeInfoWithStorage[127.0.0.1:37061,DS-c817cfbe-f58e-4f4d-94e0-14281a32ca4c,DISK]] 2023-02-13 17:53:54,943 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-13 17:53:54,943 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:54,943 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 17:53:54,943 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 17:53:54,948 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-13 17:53:54,949 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-13 17:53:54,950 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-13 17:53:54,959 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(539): loaded hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/5e57838929254d419cea807721036b05 2023-02-13 17:53:54,960 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:54,965 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5178): Found 1 recovered edits file(s) under hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-02-13 17:53:54,965 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5275): Replaying edits from hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C46373%2C1676310828881.1676310829113 2023-02-13 17:53:54,973 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5463): Applied 0, skipped 160, firstSequenceIdInLog=3, maxSequenceIdInLog=143, path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C46373%2C1676310828881.1676310829113 2023-02-13 17:53:54,974 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5085): Deleted recovered.edits file=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C46373%2C1676310828881.1676310829113 2023-02-13 17:53:54,978 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 17:53:54,984 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/143.seqid, newMaxSeqId=143, maxSeqId=1 2023-02-13 17:53:54,985 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=144; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70097630, jitterRate=0.044536083936691284}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:53:54,986 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676310834944Initializing all the Stores at 1676310834945 (+1 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676310834945Cleaning up temporary data from old regions at 1676310834975 (+30 ms)Cleaning up detritus from prior splits at 1676310834975Region opened successfully at 1676310834986 (+11 ms) 2023-02-13 17:53:54,986 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-13 17:53:54,987 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-13 17:53:54,987 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-13 17:53:54,987 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-13 17:53:54,988 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-13 17:53:54,993 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-13 17:53:54,993 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-13 17:53:54,993 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-13 17:53:54,993 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-13 17:53:54,994 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=restartTableOne 2023-02-13 17:53:54,994 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; CreateTableProcedure table=restartTableTwo 2023-02-13 17:53:54,995 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=15, state=SUCCESS; CreateTableProcedure table=restartTableThree 2023-02-13 17:53:54,995 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-02-13 17:53:54,995 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-13 17:53:54,995 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-13 17:53:54,996 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,42905,1676310828981, table=hbase:meta, region=1588230740 2023-02-13 17:53:54,997 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 3 possibly 'live' servers, and 0 'splitting'. 2023-02-13 17:53:55,000 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,42905,1676310828981 already deleted, retry=false 2023-02-13 17:53:55,000 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,42905,1676310828981 on jenkins-hbase4.apache.org,37835,1676310834816 2023-02-13 17:53:55,001 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=18, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,42905,1676310828981, splitWal=true, meta=true 2023-02-13 17:53:55,002 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1736): Scheduled ServerCrashProcedure pid=18 for jenkins-hbase4.apache.org,42905,1676310828981 (carryingMeta=true) jenkins-hbase4.apache.org,42905,1676310828981/CRASHED/regionCount=1/lock=java.util.concurrent.locks.ReentrantReadWriteLock@399c417a[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-02-13 17:53:55,003 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,39461,1676310828923 already deleted, retry=false 2023-02-13 17:53:55,003 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,39461,1676310828923 on jenkins-hbase4.apache.org,37835,1676310834816 2023-02-13 17:53:55,003 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,39461,1676310828923, splitWal=true, meta=false 2023-02-13 17:53:55,004 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1736): Scheduled ServerCrashProcedure pid=19 for jenkins-hbase4.apache.org,39461,1676310828923 (carryingMeta=false) jenkins-hbase4.apache.org,39461,1676310828923/CRASHED/regionCount=0/lock=java.util.concurrent.locks.ReentrantReadWriteLock@2ba10ab6[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-02-13 17:53:55,005 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,34875,1676310828951 already deleted, retry=false 2023-02-13 17:53:55,005 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,34875,1676310828951 on jenkins-hbase4.apache.org,37835,1676310834816 2023-02-13 17:53:55,005 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=20, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,34875,1676310828951, splitWal=true, meta=false 2023-02-13 17:53:55,006 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1736): Scheduled ServerCrashProcedure pid=20 for jenkins-hbase4.apache.org,34875,1676310828951 (carryingMeta=false) jenkins-hbase4.apache.org,34875,1676310828951/CRASHED/regionCount=0/lock=java.util.concurrent.locks.ReentrantReadWriteLock@4b41e5f0[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-02-13 17:53:55,009 INFO [RS:3;jenkins-hbase4:39461] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,37835,1676310834816 with isa=jenkins-hbase4.apache.org/172.31.14.131:39461, startcode=1676310834849 2023-02-13 17:53:55,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37835] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:53:55,010 DEBUG [RS:3;jenkins-hbase4:39461] regionserver.HRegionServer(2829): Master is not running yet 2023-02-13 17:53:55,010 WARN [RS:3;jenkins-hbase4:39461] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 200 ms and then retrying. 2023-02-13 17:53:55,016 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-13 17:53:55,017 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 17:53:55,017 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/balancer 2023-02-13 17:53:55,017 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-13 17:53:55,018 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-13 17:53:55,018 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-13 17:53:55,019 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-13 17:53:55,020 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-13 17:53:55,020 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,37835,1676310834816, sessionid=0x1004ed140fc0009, setting cluster-up flag (Was=true) 2023-02-13 17:53:55,022 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-13 17:53:55,022 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,37835,1676310834816 2023-02-13 17:53:55,025 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-13 17:53:55,026 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,37835,1676310834816 2023-02-13 17:53:55,027 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/.hbase-snapshot/.tmp 2023-02-13 17:53:55,028 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 17:53:55,028 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 17:53:55,028 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 17:53:55,028 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 17:53:55,028 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-13 17:53:55,028 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,028 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 17:53:55,028 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,030 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676310865030 2023-02-13 17:53:55,030 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-13 17:53:55,031 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-13 17:53:55,031 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-13 17:53:55,031 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-13 17:53:55,031 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-13 17:53:55,031 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-13 17:53:55,031 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:55,032 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-13 17:53:55,032 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-13 17:53:55,032 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-13 17:53:55,033 DEBUG [PEWorker-3] master.DeadServer(103): Processing jenkins-hbase4.apache.org,34875,1676310828951; numProcessing=1 2023-02-13 17:53:55,033 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-13 17:53:55,033 INFO [PEWorker-3] procedure.ServerCrashProcedure(161): Start pid=20, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,34875,1676310828951, splitWal=true, meta=false 2023-02-13 17:53:55,033 DEBUG [PEWorker-1] master.DeadServer(103): Processing jenkins-hbase4.apache.org,42905,1676310828981; numProcessing=2 2023-02-13 17:53:55,033 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-13 17:53:55,033 INFO [PEWorker-1] procedure.ServerCrashProcedure(161): Start pid=18, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,42905,1676310828981, splitWal=true, meta=true 2023-02-13 17:53:55,033 DEBUG [PEWorker-2] master.DeadServer(103): Processing jenkins-hbase4.apache.org,39461,1676310828923; numProcessing=3 2023-02-13 17:53:55,034 INFO [PEWorker-2] procedure.ServerCrashProcedure(161): Start pid=19, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,39461,1676310828923, splitWal=true, meta=false 2023-02-13 17:53:55,034 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676310835034,5,FailOnTimeoutGroup] 2023-02-13 17:53:55,034 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676310835034,5,FailOnTimeoutGroup] 2023-02-13 17:53:55,034 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:55,034 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-13 17:53:55,034 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:55,034 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:55,035 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676310834838, completionTime=-1 2023-02-13 17:53:55,035 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(801): Waiting on regionserver count=0; waited=0ms, expecting min=3 server(s), max=3 server(s), timeout=4500ms, lastChange=0ms 2023-02-13 17:53:55,037 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=18, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,42905,1676310828981, splitWal=true, meta=true, isMeta: true 2023-02-13 17:53:55,039 DEBUG [PEWorker-1] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,42905,1676310828981-splitting 2023-02-13 17:53:55,040 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,42905,1676310828981-splitting dir is empty, no logs to split. 2023-02-13 17:53:55,040 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,42905,1676310828981 WAL count=0, meta=true 2023-02-13 17:53:55,042 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,42905,1676310828981-splitting dir is empty, no logs to split. 2023-02-13 17:53:55,042 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,42905,1676310828981 WAL count=0, meta=true 2023-02-13 17:53:55,042 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,42905,1676310828981 WAL splitting is done? wals=0, meta=true 2023-02-13 17:53:55,043 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-13 17:53:55,044 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-13 17:53:55,046 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,42905,1676310828981; forceNewPlan=false, retain=false 2023-02-13 17:53:55,196 WARN [jenkins-hbase4:37835] assignment.AssignmentManager(2157): No servers available; cannot place 1 unassigned regions. 2023-02-13 17:53:55,211 INFO [RS:3;jenkins-hbase4:39461] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,37835,1676310834816 with isa=jenkins-hbase4.apache.org/172.31.14.131:39461, startcode=1676310834849 2023-02-13 17:53:55,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37835] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,39461,1676310834849 2023-02-13 17:53:55,213 DEBUG [RS:3;jenkins-hbase4:39461] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e 2023-02-13 17:53:55,213 DEBUG [RS:3;jenkins-hbase4:39461] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:36959 2023-02-13 17:53:55,213 DEBUG [RS:3;jenkins-hbase4:39461] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 17:53:55,214 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:55,215 DEBUG [RS:3;jenkins-hbase4:39461] zookeeper.ZKUtil(162): regionserver:39461-0x1004ed140fc000a, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39461,1676310834849 2023-02-13 17:53:55,215 WARN [RS:3;jenkins-hbase4:39461] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 17:53:55,215 INFO [RS:3;jenkins-hbase4:39461] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 17:53:55,216 DEBUG [RS:3;jenkins-hbase4:39461] regionserver.HRegionServer(1947): logDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,39461,1676310834849 2023-02-13 17:53:55,216 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,39461,1676310834849] 2023-02-13 17:53:55,220 DEBUG [RS:3;jenkins-hbase4:39461] zookeeper.ZKUtil(162): regionserver:39461-0x1004ed140fc000a, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39461,1676310834849 2023-02-13 17:53:55,221 DEBUG [RS:3;jenkins-hbase4:39461] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 17:53:55,221 INFO [RS:3;jenkins-hbase4:39461] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 17:53:55,223 INFO [RS:3;jenkins-hbase4:39461] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 17:53:55,223 INFO [RS:3;jenkins-hbase4:39461] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 17:53:55,223 INFO [RS:3;jenkins-hbase4:39461] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:55,223 INFO [RS:3;jenkins-hbase4:39461] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 17:53:55,225 INFO [RS:3;jenkins-hbase4:39461] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:55,225 DEBUG [RS:3;jenkins-hbase4:39461] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:39461, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,225 DEBUG [RS:3;jenkins-hbase4:39461] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:39461, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,225 DEBUG [RS:3;jenkins-hbase4:39461] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:39461, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,225 DEBUG [RS:3;jenkins-hbase4:39461] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:39461, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,225 DEBUG [RS:3;jenkins-hbase4:39461] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:39461, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,225 DEBUG [RS:3;jenkins-hbase4:39461] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:39461, corePoolSize=2, maxPoolSize=2 2023-02-13 17:53:55,225 DEBUG [RS:3;jenkins-hbase4:39461] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:39461, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,225 DEBUG [RS:3;jenkins-hbase4:39461] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:39461, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,225 DEBUG [RS:3;jenkins-hbase4:39461] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:39461, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,225 DEBUG [RS:3;jenkins-hbase4:39461] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:39461, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,226 INFO [RS:3;jenkins-hbase4:39461] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:55,226 INFO [RS:3;jenkins-hbase4:39461] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:55,226 INFO [RS:3;jenkins-hbase4:39461] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:55,235 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(801): Waiting on regionserver count=1; waited=200ms, expecting min=3 server(s), max=3 server(s), timeout=4500ms, lastChange=0ms 2023-02-13 17:53:55,239 INFO [RS:3;jenkins-hbase4:39461] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 17:53:55,239 INFO [RS:3;jenkins-hbase4:39461] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39461,1676310834849-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:55,256 INFO [RS:3;jenkins-hbase4:39461] regionserver.Replication(203): jenkins-hbase4.apache.org,39461,1676310834849 started 2023-02-13 17:53:55,256 INFO [RS:3;jenkins-hbase4:39461] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,39461,1676310834849, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:39461, sessionid=0x1004ed140fc000a 2023-02-13 17:53:55,256 DEBUG [RS:3;jenkins-hbase4:39461] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 17:53:55,256 DEBUG [RS:3;jenkins-hbase4:39461] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,39461,1676310834849 2023-02-13 17:53:55,256 DEBUG [RS:3;jenkins-hbase4:39461] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39461,1676310834849' 2023-02-13 17:53:55,256 DEBUG [RS:3;jenkins-hbase4:39461] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 17:53:55,257 DEBUG [RS:3;jenkins-hbase4:39461] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 17:53:55,268 DEBUG [RS:3;jenkins-hbase4:39461] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 17:53:55,268 DEBUG [RS:3;jenkins-hbase4:39461] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 17:53:55,268 DEBUG [RS:3;jenkins-hbase4:39461] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,39461,1676310834849 2023-02-13 17:53:55,268 DEBUG [RS:3;jenkins-hbase4:39461] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39461,1676310834849' 2023-02-13 17:53:55,268 DEBUG [RS:3;jenkins-hbase4:39461] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 17:53:55,268 DEBUG [RS:3;jenkins-hbase4:39461] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 17:53:55,269 DEBUG [RS:3;jenkins-hbase4:39461] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 17:53:55,269 INFO [RS:3;jenkins-hbase4:39461] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 17:53:55,269 INFO [RS:3;jenkins-hbase4:39461] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 17:53:55,269 INFO [Listener at localhost/40181] client.ConnectionUtils(127): regionserver/jenkins-hbase4:34875 server-side Connection retries=45 2023-02-13 17:53:55,270 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:55,270 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:55,270 INFO [Listener at localhost/40181] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 17:53:55,270 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:55,270 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 17:53:55,270 INFO [Listener at localhost/40181] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 17:53:55,271 INFO [Listener at localhost/40181] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34875 2023-02-13 17:53:55,271 INFO [Listener at localhost/40181] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 17:53:55,273 DEBUG [Listener at localhost/40181] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 17:53:55,273 INFO [Listener at localhost/40181] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:55,274 INFO [Listener at localhost/40181] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:55,276 INFO [Listener at localhost/40181] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34875 connecting to ZooKeeper ensemble=127.0.0.1:53630 2023-02-13 17:53:55,279 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:348750x0, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 17:53:55,280 DEBUG [Listener at localhost/40181] zookeeper.ZKUtil(162): regionserver:348750x0, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 17:53:55,282 DEBUG [Listener at localhost/40181] zookeeper.ZKUtil(162): regionserver:348750x0, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-13 17:53:55,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:34875-0x1004ed140fc000d connected 2023-02-13 17:53:55,284 DEBUG [Listener at localhost/40181] zookeeper.ZKUtil(164): regionserver:34875-0x1004ed140fc000d, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 17:53:55,288 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34875 2023-02-13 17:53:55,288 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34875 2023-02-13 17:53:55,288 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34875 2023-02-13 17:53:55,289 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34875 2023-02-13 17:53:55,290 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34875 2023-02-13 17:53:55,293 INFO [RS:4;jenkins-hbase4:34875] regionserver.HRegionServer(952): ClusterId : 2a4478ba-ba99-441c-8812-a0c5f19f8b8a 2023-02-13 17:53:55,294 DEBUG [RS:4;jenkins-hbase4:34875] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 17:53:55,295 DEBUG [RS:4;jenkins-hbase4:34875] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 17:53:55,296 DEBUG [RS:4;jenkins-hbase4:34875] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 17:53:55,298 DEBUG [RS:4;jenkins-hbase4:34875] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 17:53:55,299 DEBUG [RS:4;jenkins-hbase4:34875] zookeeper.ReadOnlyZKClient(139): Connect 0x682e4f41 to 127.0.0.1:53630 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:53:55,303 DEBUG [RS:4;jenkins-hbase4:34875] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66e9ac34, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:53:55,303 DEBUG [RS:4;jenkins-hbase4:34875] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62d62431, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 17:53:55,314 DEBUG [RS:4;jenkins-hbase4:34875] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:4;jenkins-hbase4:34875 2023-02-13 17:53:55,314 INFO [RS:4;jenkins-hbase4:34875] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 17:53:55,314 INFO [RS:4;jenkins-hbase4:34875] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 17:53:55,314 DEBUG [RS:4;jenkins-hbase4:34875] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 17:53:55,315 INFO [RS:4;jenkins-hbase4:34875] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,37835,1676310834816 with isa=jenkins-hbase4.apache.org/172.31.14.131:34875, startcode=1676310835269 2023-02-13 17:53:55,315 DEBUG [RS:4;jenkins-hbase4:34875] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 17:53:55,319 INFO [RS-EventLoopGroup-13-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34641, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.10 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 17:53:55,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37835] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,34875,1676310835269 2023-02-13 17:53:55,320 DEBUG [RS:4;jenkins-hbase4:34875] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e 2023-02-13 17:53:55,320 DEBUG [RS:4;jenkins-hbase4:34875] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:36959 2023-02-13 17:53:55,320 DEBUG [RS:4;jenkins-hbase4:34875] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 17:53:55,321 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:55,321 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:39461-0x1004ed140fc000a, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:55,322 DEBUG [RS:4;jenkins-hbase4:34875] zookeeper.ZKUtil(162): regionserver:34875-0x1004ed140fc000d, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34875,1676310835269 2023-02-13 17:53:55,322 WARN [RS:4;jenkins-hbase4:34875] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 17:53:55,322 INFO [RS:4;jenkins-hbase4:34875] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 17:53:55,322 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,34875,1676310835269] 2023-02-13 17:53:55,322 DEBUG [RS:4;jenkins-hbase4:34875] regionserver.HRegionServer(1947): logDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,34875,1676310835269 2023-02-13 17:53:55,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39461-0x1004ed140fc000a, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34875,1676310835269 2023-02-13 17:53:55,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39461-0x1004ed140fc000a, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39461,1676310834849 2023-02-13 17:53:55,327 DEBUG [RS:4;jenkins-hbase4:34875] zookeeper.ZKUtil(162): regionserver:34875-0x1004ed140fc000d, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34875,1676310835269 2023-02-13 17:53:55,327 DEBUG [RS:4;jenkins-hbase4:34875] zookeeper.ZKUtil(162): regionserver:34875-0x1004ed140fc000d, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39461,1676310834849 2023-02-13 17:53:55,329 DEBUG [RS:4;jenkins-hbase4:34875] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 17:53:55,329 INFO [RS:4;jenkins-hbase4:34875] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 17:53:55,331 INFO [RS:4;jenkins-hbase4:34875] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 17:53:55,331 INFO [RS:4;jenkins-hbase4:34875] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 17:53:55,331 INFO [RS:4;jenkins-hbase4:34875] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:55,332 INFO [RS:4;jenkins-hbase4:34875] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 17:53:55,333 INFO [RS:4;jenkins-hbase4:34875] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:55,334 DEBUG [RS:4;jenkins-hbase4:34875] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:34875, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,334 DEBUG [RS:4;jenkins-hbase4:34875] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:34875, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,334 DEBUG [RS:4;jenkins-hbase4:34875] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:34875, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,334 DEBUG [RS:4;jenkins-hbase4:34875] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:34875, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,334 DEBUG [RS:4;jenkins-hbase4:34875] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:34875, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,334 DEBUG [RS:4;jenkins-hbase4:34875] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:34875, corePoolSize=2, maxPoolSize=2 2023-02-13 17:53:55,334 DEBUG [RS:4;jenkins-hbase4:34875] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:34875, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,334 DEBUG [RS:4;jenkins-hbase4:34875] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:34875, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,334 DEBUG [RS:4;jenkins-hbase4:34875] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:34875, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,334 DEBUG [RS:4;jenkins-hbase4:34875] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:34875, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,335 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(801): Waiting on regionserver count=2; waited=300ms, expecting min=3 server(s), max=3 server(s), timeout=4500ms, lastChange=0ms 2023-02-13 17:53:55,335 INFO [RS:4;jenkins-hbase4:34875] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:55,335 INFO [RS:4;jenkins-hbase4:34875] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:55,335 INFO [RS:4;jenkins-hbase4:34875] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:55,347 INFO [RS:4;jenkins-hbase4:34875] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 17:53:55,347 INFO [RS:4;jenkins-hbase4:34875] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34875,1676310835269-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:55,358 INFO [RS:4;jenkins-hbase4:34875] regionserver.Replication(203): jenkins-hbase4.apache.org,34875,1676310835269 started 2023-02-13 17:53:55,358 INFO [RS:4;jenkins-hbase4:34875] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,34875,1676310835269, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:34875, sessionid=0x1004ed140fc000d 2023-02-13 17:53:55,358 DEBUG [RS:4;jenkins-hbase4:34875] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 17:53:55,358 DEBUG [RS:4;jenkins-hbase4:34875] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,34875,1676310835269 2023-02-13 17:53:55,358 DEBUG [RS:4;jenkins-hbase4:34875] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34875,1676310835269' 2023-02-13 17:53:55,358 DEBUG [RS:4;jenkins-hbase4:34875] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 17:53:55,359 DEBUG [RS:4;jenkins-hbase4:34875] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 17:53:55,359 DEBUG [RS:4;jenkins-hbase4:34875] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 17:53:55,359 DEBUG [RS:4;jenkins-hbase4:34875] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 17:53:55,359 DEBUG [RS:4;jenkins-hbase4:34875] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,34875,1676310835269 2023-02-13 17:53:55,359 DEBUG [RS:4;jenkins-hbase4:34875] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34875,1676310835269' 2023-02-13 17:53:55,359 DEBUG [RS:4;jenkins-hbase4:34875] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 17:53:55,359 DEBUG [RS:4;jenkins-hbase4:34875] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 17:53:55,360 DEBUG [RS:4;jenkins-hbase4:34875] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 17:53:55,360 INFO [RS:4;jenkins-hbase4:34875] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 17:53:55,360 INFO [RS:4;jenkins-hbase4:34875] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 17:53:55,369 INFO [Listener at localhost/40181] client.ConnectionUtils(127): regionserver/jenkins-hbase4:42905 server-side Connection retries=45 2023-02-13 17:53:55,369 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:55,369 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:55,369 INFO [Listener at localhost/40181] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 17:53:55,370 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 17:53:55,370 INFO [Listener at localhost/40181] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 17:53:55,370 INFO [Listener at localhost/40181] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 17:53:55,370 INFO [Listener at localhost/40181] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42905 2023-02-13 17:53:55,371 INFO [Listener at localhost/40181] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 17:53:55,371 INFO [RS:3;jenkins-hbase4:39461] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39461%2C1676310834849, suffix=, logDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,39461,1676310834849, archiveDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/oldWALs, maxLogs=32 2023-02-13 17:53:55,372 DEBUG [Listener at localhost/40181] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 17:53:55,373 INFO [Listener at localhost/40181] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:55,374 INFO [Listener at localhost/40181] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 17:53:55,376 INFO [Listener at localhost/40181] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:42905 connecting to ZooKeeper ensemble=127.0.0.1:53630 2023-02-13 17:53:55,379 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:429050x0, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 17:53:55,380 DEBUG [Listener at localhost/40181] zookeeper.ZKUtil(162): regionserver:429050x0, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 17:53:55,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:42905-0x1004ed140fc000f connected 2023-02-13 17:53:55,381 DEBUG [Listener at localhost/40181] zookeeper.ZKUtil(162): regionserver:42905-0x1004ed140fc000f, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-13 17:53:55,382 DEBUG [Listener at localhost/40181] zookeeper.ZKUtil(164): regionserver:42905-0x1004ed140fc000f, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 17:53:55,382 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42905 2023-02-13 17:53:55,382 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42905 2023-02-13 17:53:55,383 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42905 2023-02-13 17:53:55,384 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42905 2023-02-13 17:53:55,384 DEBUG [Listener at localhost/40181] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42905 2023-02-13 17:53:55,386 INFO [RS:5;jenkins-hbase4:42905] regionserver.HRegionServer(952): ClusterId : 2a4478ba-ba99-441c-8812-a0c5f19f8b8a 2023-02-13 17:53:55,386 DEBUG [RS:5;jenkins-hbase4:42905] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 17:53:55,389 DEBUG [RS:5;jenkins-hbase4:42905] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 17:53:55,389 DEBUG [RS:5;jenkins-hbase4:42905] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 17:53:55,392 DEBUG [RS-EventLoopGroup-14-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37727,DS-c79d1410-43a1-4b12-a2c2-dff1472abceb,DISK] 2023-02-13 17:53:55,392 DEBUG [RS-EventLoopGroup-14-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37061,DS-c817cfbe-f58e-4f4d-94e0-14281a32ca4c,DISK] 2023-02-13 17:53:55,392 DEBUG [RS-EventLoopGroup-14-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39523,DS-cf1e0e36-9598-4b66-ab37-608d409fcf08,DISK] 2023-02-13 17:53:55,393 DEBUG [RS:5;jenkins-hbase4:42905] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 17:53:55,396 DEBUG [RS:5;jenkins-hbase4:42905] zookeeper.ReadOnlyZKClient(139): Connect 0x2636a12c to 127.0.0.1:53630 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 17:53:55,396 INFO [RS:3;jenkins-hbase4:39461] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,39461,1676310834849/jenkins-hbase4.apache.org%2C39461%2C1676310834849.1676310835372 2023-02-13 17:53:55,400 DEBUG [RS:3;jenkins-hbase4:39461] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37727,DS-c79d1410-43a1-4b12-a2c2-dff1472abceb,DISK], DatanodeInfoWithStorage[127.0.0.1:39523,DS-cf1e0e36-9598-4b66-ab37-608d409fcf08,DISK], DatanodeInfoWithStorage[127.0.0.1:37061,DS-c817cfbe-f58e-4f4d-94e0-14281a32ca4c,DISK]] 2023-02-13 17:53:55,404 DEBUG [RS:5;jenkins-hbase4:42905] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d33930c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 17:53:55,405 DEBUG [RS:5;jenkins-hbase4:42905] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@175ac6ae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 17:53:55,412 DEBUG [RS:5;jenkins-hbase4:42905] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:5;jenkins-hbase4:42905 2023-02-13 17:53:55,413 INFO [RS:5;jenkins-hbase4:42905] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 17:53:55,413 INFO [RS:5;jenkins-hbase4:42905] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 17:53:55,413 DEBUG [RS:5;jenkins-hbase4:42905] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 17:53:55,413 INFO [RS:5;jenkins-hbase4:42905] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,37835,1676310834816 with isa=jenkins-hbase4.apache.org/172.31.14.131:42905, startcode=1676310835369 2023-02-13 17:53:55,413 DEBUG [RS:5;jenkins-hbase4:42905] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 17:53:55,417 INFO [RS-EventLoopGroup-13-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52885, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.11 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 17:53:55,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37835] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,42905,1676310835369 2023-02-13 17:53:55,418 DEBUG [RS:5;jenkins-hbase4:42905] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e 2023-02-13 17:53:55,418 DEBUG [RS:5;jenkins-hbase4:42905] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:36959 2023-02-13 17:53:55,418 DEBUG [RS:5;jenkins-hbase4:42905] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 17:53:55,420 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:39461-0x1004ed140fc000a, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:55,420 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:34875-0x1004ed140fc000d, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:55,420 DEBUG [RS:5;jenkins-hbase4:42905] zookeeper.ZKUtil(162): regionserver:42905-0x1004ed140fc000f, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42905,1676310835369 2023-02-13 17:53:55,420 WARN [RS:5;jenkins-hbase4:42905] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 17:53:55,420 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:55,420 INFO [RS:5;jenkins-hbase4:42905] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 17:53:55,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39461-0x1004ed140fc000a, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34875,1676310835269 2023-02-13 17:53:55,421 DEBUG [RS:5;jenkins-hbase4:42905] regionserver.HRegionServer(1947): logDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,42905,1676310835369 2023-02-13 17:53:55,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34875-0x1004ed140fc000d, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34875,1676310835269 2023-02-13 17:53:55,421 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,42905,1676310835369] 2023-02-13 17:53:55,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39461-0x1004ed140fc000a, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42905,1676310835369 2023-02-13 17:53:55,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39461-0x1004ed140fc000a, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39461,1676310834849 2023-02-13 17:53:55,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34875-0x1004ed140fc000d, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42905,1676310835369 2023-02-13 17:53:55,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34875-0x1004ed140fc000d, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39461,1676310834849 2023-02-13 17:53:55,427 DEBUG [RS:5;jenkins-hbase4:42905] zookeeper.ZKUtil(162): regionserver:42905-0x1004ed140fc000f, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34875,1676310835269 2023-02-13 17:53:55,427 DEBUG [RS:5;jenkins-hbase4:42905] zookeeper.ZKUtil(162): regionserver:42905-0x1004ed140fc000f, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42905,1676310835369 2023-02-13 17:53:55,427 DEBUG [RS:5;jenkins-hbase4:42905] zookeeper.ZKUtil(162): regionserver:42905-0x1004ed140fc000f, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39461,1676310834849 2023-02-13 17:53:55,428 DEBUG [RS:5;jenkins-hbase4:42905] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 17:53:55,428 INFO [RS:5;jenkins-hbase4:42905] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 17:53:55,429 INFO [RS:5;jenkins-hbase4:42905] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 17:53:55,430 INFO [RS:5;jenkins-hbase4:42905] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 17:53:55,430 INFO [RS:5;jenkins-hbase4:42905] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:55,430 INFO [RS:5;jenkins-hbase4:42905] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 17:53:55,431 INFO [RS:5;jenkins-hbase4:42905] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:55,431 DEBUG [RS:5;jenkins-hbase4:42905] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:42905, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,431 DEBUG [RS:5;jenkins-hbase4:42905] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:42905, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,431 DEBUG [RS:5;jenkins-hbase4:42905] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:42905, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,431 DEBUG [RS:5;jenkins-hbase4:42905] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:42905, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,431 DEBUG [RS:5;jenkins-hbase4:42905] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:42905, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,431 DEBUG [RS:5;jenkins-hbase4:42905] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:42905, corePoolSize=2, maxPoolSize=2 2023-02-13 17:53:55,431 DEBUG [RS:5;jenkins-hbase4:42905] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:42905, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,432 DEBUG [RS:5;jenkins-hbase4:42905] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:42905, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,432 DEBUG [RS:5;jenkins-hbase4:42905] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:42905, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,432 DEBUG [RS:5;jenkins-hbase4:42905] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:42905, corePoolSize=1, maxPoolSize=1 2023-02-13 17:53:55,435 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=400ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-13 17:53:55,437 WARN [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1230): hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1676310835046, server=jenkins-hbase4.apache.org,42905,1676310828981}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2023-02-13 17:53:55,437 INFO [RS:5;jenkins-hbase4:42905] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:55,437 INFO [RS:5;jenkins-hbase4:42905] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:55,437 INFO [RS:5;jenkins-hbase4:42905] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:55,447 INFO [jenkins-hbase4:37835] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 17:53:55,447 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,42905,1676310835369, state=OPENING 2023-02-13 17:53:55,449 INFO [RS:5;jenkins-hbase4:42905] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 17:53:55,449 INFO [RS:5;jenkins-hbase4:42905] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42905,1676310835369-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:55,450 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-13 17:53:55,450 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 17:53:55,450 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,42905,1676310835369}] 2023-02-13 17:53:55,461 INFO [RS:4;jenkins-hbase4:34875] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34875%2C1676310835269, suffix=, logDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,34875,1676310835269, archiveDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/oldWALs, maxLogs=32 2023-02-13 17:53:55,464 INFO [RS:5;jenkins-hbase4:42905] regionserver.Replication(203): jenkins-hbase4.apache.org,42905,1676310835369 started 2023-02-13 17:53:55,464 INFO [RS:5;jenkins-hbase4:42905] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,42905,1676310835369, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:42905, sessionid=0x1004ed140fc000f 2023-02-13 17:53:55,464 DEBUG [RS:5;jenkins-hbase4:42905] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 17:53:55,464 DEBUG [RS:5;jenkins-hbase4:42905] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,42905,1676310835369 2023-02-13 17:53:55,464 DEBUG [RS:5;jenkins-hbase4:42905] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42905,1676310835369' 2023-02-13 17:53:55,464 DEBUG [RS:5;jenkins-hbase4:42905] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 17:53:55,464 INFO [Listener at localhost/40181] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:55,464 DEBUG [RS:5;jenkins-hbase4:42905] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 17:53:55,465 DEBUG [RS:5;jenkins-hbase4:42905] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 17:53:55,465 DEBUG [RS:5;jenkins-hbase4:42905] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 17:53:55,465 DEBUG [RS:5;jenkins-hbase4:42905] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,42905,1676310835369 2023-02-13 17:53:55,465 DEBUG [RS:5;jenkins-hbase4:42905] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42905,1676310835369' 2023-02-13 17:53:55,465 DEBUG [RS:5;jenkins-hbase4:42905] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 17:53:55,465 DEBUG [RS:5;jenkins-hbase4:42905] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 17:53:55,466 DEBUG [RS:5;jenkins-hbase4:42905] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 17:53:55,466 INFO [RS:5;jenkins-hbase4:42905] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 17:53:55,466 INFO [RS:5;jenkins-hbase4:42905] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 17:53:55,466 INFO [RS-EventLoopGroup-16-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47364, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:53:55,466 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=42905] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Get size: 90 connection: 172.31.14.131:47364 deadline: 1676310895466, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,42905,1676310835369 2023-02-13 17:53:55,475 DEBUG [RS-EventLoopGroup-15-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39523,DS-cf1e0e36-9598-4b66-ab37-608d409fcf08,DISK] 2023-02-13 17:53:55,475 DEBUG [RS-EventLoopGroup-15-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37727,DS-c79d1410-43a1-4b12-a2c2-dff1472abceb,DISK] 2023-02-13 17:53:55,476 DEBUG [RS-EventLoopGroup-15-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37061,DS-c817cfbe-f58e-4f4d-94e0-14281a32ca4c,DISK] 2023-02-13 17:53:55,477 INFO [RS:4;jenkins-hbase4:34875] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,34875,1676310835269/jenkins-hbase4.apache.org%2C34875%2C1676310835269.1676310835462 2023-02-13 17:53:55,478 DEBUG [RS:4;jenkins-hbase4:34875] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39523,DS-cf1e0e36-9598-4b66-ab37-608d409fcf08,DISK], DatanodeInfoWithStorage[127.0.0.1:37727,DS-c79d1410-43a1-4b12-a2c2-dff1472abceb,DISK], DatanodeInfoWithStorage[127.0.0.1:37061,DS-c817cfbe-f58e-4f4d-94e0-14281a32ca4c,DISK]] 2023-02-13 17:53:55,568 INFO [RS:5;jenkins-hbase4:42905] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42905%2C1676310835369, suffix=, logDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,42905,1676310835369, archiveDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/oldWALs, maxLogs=32 2023-02-13 17:53:55,568 WARN [ReadOnlyZKClient-127.0.0.1:53630@0x410494f3] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-02-13 17:53:55,569 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=42905] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Get size: 90 connection: 172.31.14.131:47364 deadline: 1676310895569, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,42905,1676310835369 2023-02-13 17:53:55,583 DEBUG [RS-EventLoopGroup-16-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39523,DS-cf1e0e36-9598-4b66-ab37-608d409fcf08,DISK] 2023-02-13 17:53:55,583 DEBUG [RS-EventLoopGroup-16-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37727,DS-c79d1410-43a1-4b12-a2c2-dff1472abceb,DISK] 2023-02-13 17:53:55,583 DEBUG [RS-EventLoopGroup-16-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37061,DS-c817cfbe-f58e-4f4d-94e0-14281a32ca4c,DISK] 2023-02-13 17:53:55,588 INFO [RS:5;jenkins-hbase4:42905] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,42905,1676310835369/jenkins-hbase4.apache.org%2C42905%2C1676310835369.1676310835568 2023-02-13 17:53:55,590 DEBUG [RS:5;jenkins-hbase4:42905] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37061,DS-c817cfbe-f58e-4f4d-94e0-14281a32ca4c,DISK], DatanodeInfoWithStorage[127.0.0.1:37727,DS-c79d1410-43a1-4b12-a2c2-dff1472abceb,DISK], DatanodeInfoWithStorage[127.0.0.1:39523,DS-cf1e0e36-9598-4b66-ab37-608d409fcf08,DISK]] 2023-02-13 17:53:55,607 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,42905,1676310835369 2023-02-13 17:53:55,607 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 17:53:55,608 INFO [RS-EventLoopGroup-16-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47380, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.8 (auth:SIMPLE), service=AdminService 2023-02-13 17:53:55,612 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:42905-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-13 17:53:55,612 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:42905-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 17:53:55,614 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:42905-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42905%2C1676310835369.meta, suffix=.meta, logDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,42905,1676310835369, archiveDir=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/oldWALs, maxLogs=32 2023-02-13 17:53:55,628 DEBUG [RS-EventLoopGroup-16-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37061,DS-c817cfbe-f58e-4f4d-94e0-14281a32ca4c,DISK] 2023-02-13 17:53:55,628 DEBUG [RS-EventLoopGroup-16-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37727,DS-c79d1410-43a1-4b12-a2c2-dff1472abceb,DISK] 2023-02-13 17:53:55,628 DEBUG [RS-EventLoopGroup-16-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39523,DS-cf1e0e36-9598-4b66-ab37-608d409fcf08,DISK] 2023-02-13 17:53:55,632 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:42905-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,42905,1676310835369/jenkins-hbase4.apache.org%2C42905%2C1676310835369.meta.1676310835614.meta 2023-02-13 17:53:55,632 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42905-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39523,DS-cf1e0e36-9598-4b66-ab37-608d409fcf08,DISK], DatanodeInfoWithStorage[127.0.0.1:37727,DS-c79d1410-43a1-4b12-a2c2-dff1472abceb,DISK], DatanodeInfoWithStorage[127.0.0.1:37061,DS-c817cfbe-f58e-4f4d-94e0-14281a32ca4c,DISK]] 2023-02-13 17:53:55,632 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-13 17:53:55,633 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42905-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-13 17:53:55,633 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-13 17:53:55,634 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:42905-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-13 17:53:55,634 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42905-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-13 17:53:55,634 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:55,634 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-13 17:53:55,634 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-13 17:53:55,635 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 17:53:55,637 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/info 2023-02-13 17:53:55,637 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/info 2023-02-13 17:53:55,637 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 17:53:55,647 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/info/0174fadab1e947fa930999dff816966f 2023-02-13 17:53:55,648 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:55,648 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 17:53:55,649 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/rep_barrier 2023-02-13 17:53:55,649 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/rep_barrier 2023-02-13 17:53:55,649 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 17:53:55,650 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:55,650 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 17:53:55,651 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/table 2023-02-13 17:53:55,651 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/table 2023-02-13 17:53:55,652 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 17:53:55,659 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/table/c5d2fb40a5f947c88b9739f47ec9436c 2023-02-13 17:53:55,659 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:55,660 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740 2023-02-13 17:53:55,662 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740 2023-02-13 17:53:55,664 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42905-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 17:53:55,666 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 17:53:55,667 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=28; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72105688, jitterRate=0.07445847988128662}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 17:53:55,667 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676310835634Writing region info on filesystem at 1676310835634Initializing all the Stores at 1676310835635 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676310835635Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676310835635Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676310835635Cleaning up temporary data from old regions at 1676310835663 (+28 ms)Cleaning up detritus from prior splits at 1676310835663Running coprocessor post-open hooks at 1676310835667 (+4 ms)Region opened successfully at 1676310835667 2023-02-13 17:53:55,668 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:42905-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=22, masterSystemTime=1676310835607 2023-02-13 17:53:55,672 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42905-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-13 17:53:55,672 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:42905-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-13 17:53:55,673 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,42905,1676310835369, state=OPEN 2023-02-13 17:53:55,674 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-13 17:53:55,675 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 17:53:55,677 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-02-13 17:53:55,677 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,42905,1676310835369 in 225 msec 2023-02-13 17:53:55,679 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=18 2023-02-13 17:53:55,680 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 634 msec 2023-02-13 17:53:55,785 WARN [RPCClient-NioEventLoopGroup-6-6] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:46373 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:46373 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 17:53:55,786 WARN [Listener at localhost/40181] client.ConnectionImplementation(411): Checking master connection java.net.ConnectException: Call to address=jenkins-hbase4.apache.org/172.31.14.131:46373 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:46373 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:186) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:386) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:417) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:413) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:99) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:398) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:368) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1428) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:396) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:913) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:195) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$300(NettyRpcConnection.java:76) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:296) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:287) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:46373 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 17:53:55,786 DEBUG [RPCClient-NioEventLoopGroup-6-6] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:46373 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:46373 2023-02-13 17:53:55,787 DEBUG [Listener at localhost/40181] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:46373 this server is in the failed servers list 2023-02-13 17:53:55,787 WARN [Listener at localhost/40181] client.ConnectionImplementation(411): Checking master connection org.apache.hadoop.hbase.ipc.FailedServerException: Call to address=jenkins-hbase4.apache.org/172.31.14.131:46373 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: jenkins-hbase4.apache.org/172.31.14.131:46373 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:236) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:386) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:417) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:413) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:430) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:326) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:582) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1174) at org.apache.hadoop.hbase.client.ConnectionImplementation.lambda$masterConnectionStateSupplier$1(ConnectionImplementation.java:404) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1945) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1383) at org.apache.hadoop.hbase.client.ConnectionImplementation.getMaster(ConnectionImplementation.java:1373) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:57) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:102) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:2954) at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:542) at org.apache.hadoop.hbase.client.HTable.getDescriptor(HTable.java:228) at org.apache.hadoop.hbase.HBaseTestingUtility$9.evaluate(HBaseTestingUtility.java:3983) at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:183) at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:134) at org.apache.hadoop.hbase.HBaseCommonTestingUtility.waitFor(HBaseCommonTestingUtility.java:234) at org.apache.hadoop.hbase.HBaseTestingUtility.waitTableAvailable(HBaseTestingUtility.java:3136) at org.apache.hadoop.hbase.HBaseTestingUtility.waitTableAvailable(HBaseTestingUtility.java:3121) at org.apache.hadoop.hbase.master.TestRetainAssignmentOnRestart.testRetainAssignmentOnClusterRestart(TestRetainAssignmentOnRestart.java:108) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: jenkins-hbase4.apache.org/172.31.14.131:46373 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:350) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:427) ... 46 more 2023-02-13 17:53:55,788 DEBUG [Listener at localhost/40181] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-13 17:53:55,791 INFO [RS-EventLoopGroup-13-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46056, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-13 17:53:55,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37835] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:53:55,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37835] ipc.CallRunner(144): callId: 50 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:46056 deadline: 1676310895792, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 17:53:55,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37835] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:53:55,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37835] ipc.CallRunner(144): callId: 52 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:46056 deadline: 1676310895893, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 17:53:56,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37835] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:53:56,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37835] ipc.CallRunner(144): callId: 54 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:46056 deadline: 1676310896096, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 17:53:56,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37835] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:53:56,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37835] ipc.CallRunner(144): callId: 56 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:46056 deadline: 1676310896400, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 17:53:56,437 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-13 17:53:56,441 DEBUG [hconnection-0x48b246ac-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:53:56,442 INFO [RS-EventLoopGroup-16-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47392, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.8 (auth:SIMPLE), service=ClientService 2023-02-13 17:53:56,445 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=28d66b4066b494c8b8644327cd7fdb67, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,34875,1676310828951, regionLocation=jenkins-hbase4.apache.org,34875,1676310828951, openSeqNum=2 2023-02-13 17:53:56,445 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=fbc323e3b53470e4b5a74c9688484c26, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,42905,1676310828981, regionLocation=jenkins-hbase4.apache.org,42905,1676310828981, openSeqNum=2 2023-02-13 17:53:56,445 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=d92fcd787d4162ea4435359fae28fa7d, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,34875,1676310828951, regionLocation=jenkins-hbase4.apache.org,34875,1676310828951, openSeqNum=2 2023-02-13 17:53:56,445 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=52ceb95a47ba04015d52c77bc1874032, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,42905,1676310828981, regionLocation=jenkins-hbase4.apache.org,42905,1676310828981, openSeqNum=2 2023-02-13 17:53:56,445 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-13 17:53:56,446 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676310896446 2023-02-13 17:53:56,446 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676310956446 2023-02-13 17:53:56,446 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 8 msec 2023-02-13 17:53:56,465 INFO [PEWorker-5] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,42905,1676310828981 had 3 regions 2023-02-13 17:53:56,465 INFO [PEWorker-3] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,34875,1676310828951 had 2 regions 2023-02-13 17:53:56,465 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37835,1676310834816-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:56,465 INFO [PEWorker-2] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,39461,1676310828923 had 0 regions 2023-02-13 17:53:56,465 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37835,1676310834816-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:56,465 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37835,1676310834816-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:56,465 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:37835, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:56,465 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-13 17:53:56,465 WARN [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1230): hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67. is NOT online; state={28d66b4066b494c8b8644327cd7fdb67 state=OPEN, ts=1676310836445, server=jenkins-hbase4.apache.org,34875,1676310828951}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2023-02-13 17:53:56,466 INFO [PEWorker-3] procedure.ServerCrashProcedure(300): Splitting WALs pid=20, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,34875,1676310828951, splitWal=true, meta=false, isMeta: false 2023-02-13 17:53:56,467 INFO [PEWorker-2] procedure.ServerCrashProcedure(300): Splitting WALs pid=19, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,39461,1676310828923, splitWal=true, meta=false, isMeta: false 2023-02-13 17:53:56,467 INFO [PEWorker-5] procedure.ServerCrashProcedure(300): Splitting WALs pid=18, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,42905,1676310828981, splitWal=true, meta=true, isMeta: false 2023-02-13 17:53:56,468 DEBUG [PEWorker-3] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,34875,1676310828951-splitting 2023-02-13 17:53:56,470 INFO [PEWorker-3] master.SplitLogManager(171): hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,34875,1676310828951-splitting dir is empty, no logs to split. 2023-02-13 17:53:56,470 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase4.apache.org,34875,1676310828951 WAL count=0, meta=false 2023-02-13 17:53:56,470 DEBUG [PEWorker-2] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,39461,1676310828923-splitting 2023-02-13 17:53:56,470 WARN [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(172): unknown_server=jenkins-hbase4.apache.org,34875,1676310828951/hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67., unknown_server=jenkins-hbase4.apache.org,42905,1676310828981/restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26., unknown_server=jenkins-hbase4.apache.org,34875,1676310828951/restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d., unknown_server=jenkins-hbase4.apache.org,42905,1676310828981/restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032. 2023-02-13 17:53:56,471 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,39461,1676310828923-splitting dir is empty, no logs to split. 2023-02-13 17:53:56,471 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,39461,1676310828923 WAL count=0, meta=false 2023-02-13 17:53:56,472 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,42905,1676310828981-splitting dir is empty, no logs to split. 2023-02-13 17:53:56,472 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,42905,1676310828981 WAL count=0, meta=false 2023-02-13 17:53:56,474 INFO [PEWorker-3] master.SplitLogManager(171): hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,34875,1676310828951-splitting dir is empty, no logs to split. 2023-02-13 17:53:56,474 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase4.apache.org,34875,1676310828951 WAL count=0, meta=false 2023-02-13 17:53:56,474 DEBUG [PEWorker-3] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,34875,1676310828951 WAL splitting is done? wals=0, meta=false 2023-02-13 17:53:56,474 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,39461,1676310828923-splitting dir is empty, no logs to split. 2023-02-13 17:53:56,475 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,39461,1676310828923 WAL count=0, meta=false 2023-02-13 17:53:56,475 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,39461,1676310828923 WAL splitting is done? wals=0, meta=false 2023-02-13 17:53:56,476 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,42905,1676310828981-splitting dir is empty, no logs to split. 2023-02-13 17:53:56,476 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,42905,1676310828981 WAL count=0, meta=false 2023-02-13 17:53:56,476 DEBUG [PEWorker-5] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,42905,1676310828981 WAL splitting is done? wals=0, meta=false 2023-02-13 17:53:56,477 INFO [PEWorker-3] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase4.apache.org,34875,1676310828951 failed, ignore...File hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,34875,1676310828951-splitting does not exist. 2023-02-13 17:53:56,478 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=fbc323e3b53470e4b5a74c9688484c26, ASSIGN}, {pid=24, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=52ceb95a47ba04015d52c77bc1874032, ASSIGN}] 2023-02-13 17:53:56,478 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=d92fcd787d4162ea4435359fae28fa7d, ASSIGN}, {pid=26, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=28d66b4066b494c8b8644327cd7fdb67, ASSIGN}] 2023-02-13 17:53:56,478 INFO [PEWorker-2] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase4.apache.org,39461,1676310828923 failed, ignore...File hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/WALs/jenkins-hbase4.apache.org,39461,1676310828923-splitting does not exist. 2023-02-13 17:53:56,479 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=23, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=fbc323e3b53470e4b5a74c9688484c26, ASSIGN 2023-02-13 17:53:56,479 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=52ceb95a47ba04015d52c77bc1874032, ASSIGN 2023-02-13 17:53:56,480 INFO [PEWorker-2] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,39461,1676310828923 after splitting done 2023-02-13 17:53:56,480 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42905,1676310828981 2023-02-13 17:53:56,480 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42905,1676310828981 2023-02-13 17:53:56,480 DEBUG [PEWorker-2] master.DeadServer(114): Removed jenkins-hbase4.apache.org,39461,1676310828923 from processing; numProcessing=2 2023-02-13 17:53:56,481 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=24, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableTwo, region=52ceb95a47ba04015d52c77bc1874032, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,42905,1676310828981; forceNewPlan=false, retain=true 2023-02-13 17:53:56,480 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=23, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableOne, region=fbc323e3b53470e4b5a74c9688484c26, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,42905,1676310828981; forceNewPlan=false, retain=true 2023-02-13 17:53:56,482 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,39461,1676310828923, splitWal=true, meta=false in 1.4780 sec 2023-02-13 17:53:56,485 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=28d66b4066b494c8b8644327cd7fdb67, ASSIGN 2023-02-13 17:53:56,485 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=d92fcd787d4162ea4435359fae28fa7d, ASSIGN 2023-02-13 17:53:56,486 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,34875,1676310828951 2023-02-13 17:53:56,486 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=26, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=28d66b4066b494c8b8644327cd7fdb67, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,34875,1676310828951; forceNewPlan=false, retain=true 2023-02-13 17:53:56,486 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,34875,1676310828951 2023-02-13 17:53:56,486 INFO [jenkins-hbase4:37835] balancer.BaseLoadBalancer(1545): Reassigned 3 regions. 3 retained the pre-restart assignment. 2023-02-13 17:53:56,486 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=25, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableThree, region=d92fcd787d4162ea4435359fae28fa7d, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,34875,1676310828951; forceNewPlan=false, retain=true 2023-02-13 17:53:56,488 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=52ceb95a47ba04015d52c77bc1874032, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42905,1676310835369 2023-02-13 17:53:56,488 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676310836488"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676310836488"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676310836488"}]},"ts":"1676310836488"} 2023-02-13 17:53:56,488 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=fbc323e3b53470e4b5a74c9688484c26, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42905,1676310835369 2023-02-13 17:53:56,489 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676310836488"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676310836488"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676310836488"}]},"ts":"1676310836488"} 2023-02-13 17:53:56,489 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=28d66b4066b494c8b8644327cd7fdb67, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34875,1676310835269 2023-02-13 17:53:56,489 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676310836489"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676310836489"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676310836489"}]},"ts":"1676310836489"} 2023-02-13 17:53:56,491 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=24, state=RUNNABLE; OpenRegionProcedure 52ceb95a47ba04015d52c77bc1874032, server=jenkins-hbase4.apache.org,42905,1676310835369}] 2023-02-13 17:53:56,493 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=23, state=RUNNABLE; OpenRegionProcedure fbc323e3b53470e4b5a74c9688484c26, server=jenkins-hbase4.apache.org,42905,1676310835369}] 2023-02-13 17:53:56,493 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=26, state=RUNNABLE; OpenRegionProcedure 28d66b4066b494c8b8644327cd7fdb67, server=jenkins-hbase4.apache.org,34875,1676310835269}] 2023-02-13 17:53:56,638 INFO [jenkins-hbase4:37835] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 17:53:56,639 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=d92fcd787d4162ea4435359fae28fa7d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34875,1676310835269 2023-02-13 17:53:56,640 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676310836639"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676310836639"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676310836639"}]},"ts":"1676310836639"} 2023-02-13 17:53:56,642 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=25, state=RUNNABLE; OpenRegionProcedure d92fcd787d4162ea4435359fae28fa7d, server=jenkins-hbase4.apache.org,34875,1676310835269}] 2023-02-13 17:53:56,647 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,34875,1676310835269 2023-02-13 17:53:56,647 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 17:53:56,648 INFO [RS-EventLoopGroup-15-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49516, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.8 (auth:SIMPLE), service=AdminService 2023-02-13 17:53:56,649 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] handler.AssignRegionHandler(128): Open restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032. 2023-02-13 17:53:56,649 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(7850): Opening region: {ENCODED => 52ceb95a47ba04015d52c77bc1874032, NAME => 'restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032.', STARTKEY => '', ENDKEY => ''} 2023-02-13 17:53:56,650 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableTwo 52ceb95a47ba04015d52c77bc1874032 2023-02-13 17:53:56,650 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:56,650 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(7890): checking encryption for 52ceb95a47ba04015d52c77bc1874032 2023-02-13 17:53:56,650 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(7893): checking classloading for 52ceb95a47ba04015d52c77bc1874032 2023-02-13 17:53:56,651 INFO [StoreOpener-52ceb95a47ba04015d52c77bc1874032-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region 52ceb95a47ba04015d52c77bc1874032 2023-02-13 17:53:56,652 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:34875-0] handler.AssignRegionHandler(128): Open restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d. 2023-02-13 17:53:56,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(7850): Opening region: {ENCODED => d92fcd787d4162ea4435359fae28fa7d, NAME => 'restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d.', STARTKEY => '', ENDKEY => ''} 2023-02-13 17:53:56,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableThree d92fcd787d4162ea4435359fae28fa7d 2023-02-13 17:53:56,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(865): Instantiated restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:56,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(7890): checking encryption for d92fcd787d4162ea4435359fae28fa7d 2023-02-13 17:53:56,653 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(7893): checking classloading for d92fcd787d4162ea4435359fae28fa7d 2023-02-13 17:53:56,653 DEBUG [StoreOpener-52ceb95a47ba04015d52c77bc1874032-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableTwo/52ceb95a47ba04015d52c77bc1874032/family 2023-02-13 17:53:56,653 DEBUG [StoreOpener-52ceb95a47ba04015d52c77bc1874032-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableTwo/52ceb95a47ba04015d52c77bc1874032/family 2023-02-13 17:53:56,653 INFO [StoreOpener-52ceb95a47ba04015d52c77bc1874032-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 52ceb95a47ba04015d52c77bc1874032 columnFamilyName family 2023-02-13 17:53:56,654 INFO [StoreOpener-d92fcd787d4162ea4435359fae28fa7d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region d92fcd787d4162ea4435359fae28fa7d 2023-02-13 17:53:56,654 INFO [StoreOpener-52ceb95a47ba04015d52c77bc1874032-1] regionserver.HStore(310): Store=52ceb95a47ba04015d52c77bc1874032/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:56,655 DEBUG [StoreOpener-d92fcd787d4162ea4435359fae28fa7d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableThree/d92fcd787d4162ea4435359fae28fa7d/family 2023-02-13 17:53:56,655 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableTwo/52ceb95a47ba04015d52c77bc1874032 2023-02-13 17:53:56,655 DEBUG [StoreOpener-d92fcd787d4162ea4435359fae28fa7d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableThree/d92fcd787d4162ea4435359fae28fa7d/family 2023-02-13 17:53:56,655 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:34875-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67. 2023-02-13 17:53:56,655 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(7850): Opening region: {ENCODED => 28d66b4066b494c8b8644327cd7fdb67, NAME => 'hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67.', STARTKEY => '', ENDKEY => ''} 2023-02-13 17:53:56,655 INFO [StoreOpener-d92fcd787d4162ea4435359fae28fa7d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d92fcd787d4162ea4435359fae28fa7d columnFamilyName family 2023-02-13 17:53:56,655 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 28d66b4066b494c8b8644327cd7fdb67 2023-02-13 17:53:56,655 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:56,656 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(7890): checking encryption for 28d66b4066b494c8b8644327cd7fdb67 2023-02-13 17:53:56,656 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(7893): checking classloading for 28d66b4066b494c8b8644327cd7fdb67 2023-02-13 17:53:56,656 INFO [StoreOpener-d92fcd787d4162ea4435359fae28fa7d-1] regionserver.HStore(310): Store=d92fcd787d4162ea4435359fae28fa7d/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:56,656 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableTwo/52ceb95a47ba04015d52c77bc1874032 2023-02-13 17:53:56,657 INFO [StoreOpener-28d66b4066b494c8b8644327cd7fdb67-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 28d66b4066b494c8b8644327cd7fdb67 2023-02-13 17:53:56,657 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableThree/d92fcd787d4162ea4435359fae28fa7d 2023-02-13 17:53:56,658 DEBUG [StoreOpener-28d66b4066b494c8b8644327cd7fdb67-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/namespace/28d66b4066b494c8b8644327cd7fdb67/info 2023-02-13 17:53:56,658 DEBUG [StoreOpener-28d66b4066b494c8b8644327cd7fdb67-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/namespace/28d66b4066b494c8b8644327cd7fdb67/info 2023-02-13 17:53:56,658 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableThree/d92fcd787d4162ea4435359fae28fa7d 2023-02-13 17:53:56,658 INFO [StoreOpener-28d66b4066b494c8b8644327cd7fdb67-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 28d66b4066b494c8b8644327cd7fdb67 columnFamilyName info 2023-02-13 17:53:56,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1054): writing seq id for 52ceb95a47ba04015d52c77bc1874032 2023-02-13 17:53:56,661 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1071): Opened 52ceb95a47ba04015d52c77bc1874032; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70571854, jitterRate=0.051602572202682495}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:53:56,661 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(964): Region open journal for 52ceb95a47ba04015d52c77bc1874032: Running coprocessor pre-open hook at 1676310836650Writing region info on filesystem at 1676310836650Initializing all the Stores at 1676310836651 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676310836651Cleaning up temporary data from old regions at 1676310836657 (+6 ms)Cleaning up detritus from prior splits at 1676310836657Running coprocessor post-open hooks at 1676310836661 (+4 ms)Region opened successfully at 1676310836661 2023-02-13 17:53:56,661 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(1054): writing seq id for d92fcd787d4162ea4435359fae28fa7d 2023-02-13 17:53:56,662 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032., pid=27, masterSystemTime=1676310836646 2023-02-13 17:53:56,662 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(1071): Opened d92fcd787d4162ea4435359fae28fa7d; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66689826, jitterRate=-0.006244152784347534}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:53:56,663 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(964): Region open journal for d92fcd787d4162ea4435359fae28fa7d: Running coprocessor pre-open hook at 1676310836653Writing region info on filesystem at 1676310836653Initializing all the Stores at 1676310836653Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676310836653Cleaning up temporary data from old regions at 1676310836658 (+5 ms)Cleaning up detritus from prior splits at 1676310836659 (+1 ms)Running coprocessor post-open hooks at 1676310836663 (+4 ms)Region opened successfully at 1676310836663 2023-02-13 17:53:56,664 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032. 2023-02-13 17:53:56,664 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] handler.AssignRegionHandler(156): Opened restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032. 2023-02-13 17:53:56,665 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] handler.AssignRegionHandler(128): Open restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26. 2023-02-13 17:53:56,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(7850): Opening region: {ENCODED => fbc323e3b53470e4b5a74c9688484c26, NAME => 'restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26.', STARTKEY => '', ENDKEY => ''} 2023-02-13 17:53:56,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableOne fbc323e3b53470e4b5a74c9688484c26 2023-02-13 17:53:56,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 17:53:56,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(7890): checking encryption for fbc323e3b53470e4b5a74c9688484c26 2023-02-13 17:53:56,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(7893): checking classloading for fbc323e3b53470e4b5a74c9688484c26 2023-02-13 17:53:56,665 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d., pid=30, masterSystemTime=1676310836647 2023-02-13 17:53:56,665 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=52ceb95a47ba04015d52c77bc1874032, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42905,1676310835369 2023-02-13 17:53:56,669 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676310836665"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676310836665"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676310836665"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676310836665"}]},"ts":"1676310836665"} 2023-02-13 17:53:56,669 DEBUG [StoreOpener-28d66b4066b494c8b8644327cd7fdb67-1] regionserver.HStore(539): loaded hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/namespace/28d66b4066b494c8b8644327cd7fdb67/info/438f5ae286fa473e94b8c90935fd5fb1 2023-02-13 17:53:56,670 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d. 2023-02-13 17:53:56,671 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=d92fcd787d4162ea4435359fae28fa7d, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34875,1676310835269 2023-02-13 17:53:56,671 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676310836671"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676310836671"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676310836671"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676310836671"}]},"ts":"1676310836671"} 2023-02-13 17:53:56,672 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:34875-0] handler.AssignRegionHandler(156): Opened restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d. 2023-02-13 17:53:56,673 INFO [StoreOpener-28d66b4066b494c8b8644327cd7fdb67-1] regionserver.HStore(310): Store=28d66b4066b494c8b8644327cd7fdb67/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:56,673 INFO [StoreOpener-fbc323e3b53470e4b5a74c9688484c26-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region fbc323e3b53470e4b5a74c9688484c26 2023-02-13 17:53:56,674 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/namespace/28d66b4066b494c8b8644327cd7fdb67 2023-02-13 17:53:56,676 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/namespace/28d66b4066b494c8b8644327cd7fdb67 2023-02-13 17:53:56,676 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=24 2023-02-13 17:53:56,677 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=24, state=SUCCESS; OpenRegionProcedure 52ceb95a47ba04015d52c77bc1874032, server=jenkins-hbase4.apache.org,42905,1676310835369 in 183 msec 2023-02-13 17:53:56,678 DEBUG [StoreOpener-fbc323e3b53470e4b5a74c9688484c26-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableOne/fbc323e3b53470e4b5a74c9688484c26/family 2023-02-13 17:53:56,678 DEBUG [StoreOpener-fbc323e3b53470e4b5a74c9688484c26-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableOne/fbc323e3b53470e4b5a74c9688484c26/family 2023-02-13 17:53:56,679 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=25 2023-02-13 17:53:56,679 INFO [StoreOpener-fbc323e3b53470e4b5a74c9688484c26-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fbc323e3b53470e4b5a74c9688484c26 columnFamilyName family 2023-02-13 17:53:56,679 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=25, state=SUCCESS; OpenRegionProcedure d92fcd787d4162ea4435359fae28fa7d, server=jenkins-hbase4.apache.org,34875,1676310835269 in 33 msec 2023-02-13 17:53:56,679 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=restartTableTwo, region=52ceb95a47ba04015d52c77bc1874032, ASSIGN in 199 msec 2023-02-13 17:53:56,679 INFO [StoreOpener-fbc323e3b53470e4b5a74c9688484c26-1] regionserver.HStore(310): Store=fbc323e3b53470e4b5a74c9688484c26/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 17:53:56,680 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(1054): writing seq id for 28d66b4066b494c8b8644327cd7fdb67 2023-02-13 17:53:56,680 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=restartTableThree, region=d92fcd787d4162ea4435359fae28fa7d, ASSIGN in 201 msec 2023-02-13 17:53:56,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableOne/fbc323e3b53470e4b5a74c9688484c26 2023-02-13 17:53:56,681 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(1071): Opened 28d66b4066b494c8b8644327cd7fdb67; next sequenceid=10; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66970585, jitterRate=-0.00206051766872406}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:53:56,681 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(964): Region open journal for 28d66b4066b494c8b8644327cd7fdb67: Running coprocessor pre-open hook at 1676310836656Writing region info on filesystem at 1676310836656Initializing all the Stores at 1676310836656Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676310836656Cleaning up temporary data from old regions at 1676310836676 (+20 ms)Cleaning up detritus from prior splits at 1676310836677 (+1 ms)Running coprocessor post-open hooks at 1676310836681 (+4 ms)Region opened successfully at 1676310836681 2023-02-13 17:53:56,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableOne/fbc323e3b53470e4b5a74c9688484c26 2023-02-13 17:53:56,682 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67., pid=29, masterSystemTime=1676310836647 2023-02-13 17:53:56,685 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67. 2023-02-13 17:53:56,685 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:34875-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67. 2023-02-13 17:53:56,685 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=28d66b4066b494c8b8644327cd7fdb67, regionState=OPEN, openSeqNum=10, regionLocation=jenkins-hbase4.apache.org,34875,1676310835269 2023-02-13 17:53:56,685 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1054): writing seq id for fbc323e3b53470e4b5a74c9688484c26 2023-02-13 17:53:56,685 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676310836685"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676310836685"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676310836685"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676310836685"}]},"ts":"1676310836685"} 2023-02-13 17:53:56,686 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1071): Opened fbc323e3b53470e4b5a74c9688484c26; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66645504, jitterRate=-0.00690460205078125}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 17:53:56,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(964): Region open journal for fbc323e3b53470e4b5a74c9688484c26: Running coprocessor pre-open hook at 1676310836665Writing region info on filesystem at 1676310836665Initializing all the Stores at 1676310836670 (+5 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676310836670Cleaning up temporary data from old regions at 1676310836682 (+12 ms)Cleaning up detritus from prior splits at 1676310836683 (+1 ms)Running coprocessor post-open hooks at 1676310836686 (+3 ms)Region opened successfully at 1676310836686 2023-02-13 17:53:56,687 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26., pid=28, masterSystemTime=1676310836646 2023-02-13 17:53:56,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26. 2023-02-13 17:53:56,688 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:42905-0] handler.AssignRegionHandler(156): Opened restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26. 2023-02-13 17:53:56,689 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=fbc323e3b53470e4b5a74c9688484c26, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42905,1676310835369 2023-02-13 17:53:56,689 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=26 2023-02-13 17:53:56,689 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=26, state=SUCCESS; OpenRegionProcedure 28d66b4066b494c8b8644327cd7fdb67, server=jenkins-hbase4.apache.org,34875,1676310835269 in 194 msec 2023-02-13 17:53:56,689 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676310836689"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676310836689"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676310836689"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676310836689"}]},"ts":"1676310836689"} 2023-02-13 17:53:56,691 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=20 2023-02-13 17:53:56,691 INFO [PEWorker-3] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,34875,1676310828951 after splitting done 2023-02-13 17:53:56,691 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=28d66b4066b494c8b8644327cd7fdb67, ASSIGN in 211 msec 2023-02-13 17:53:56,692 DEBUG [PEWorker-3] master.DeadServer(114): Removed jenkins-hbase4.apache.org,34875,1676310828951 from processing; numProcessing=1 2023-02-13 17:53:56,693 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=28, resume processing ppid=23 2023-02-13 17:53:56,693 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=23, state=SUCCESS; OpenRegionProcedure fbc323e3b53470e4b5a74c9688484c26, server=jenkins-hbase4.apache.org,42905,1676310835369 in 198 msec 2023-02-13 17:53:56,694 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,34875,1676310828951, splitWal=true, meta=false in 1.6870 sec 2023-02-13 17:53:56,695 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=18 2023-02-13 17:53:56,695 INFO [PEWorker-2] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,42905,1676310828981 after splitting done 2023-02-13 17:53:56,695 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=restartTableOne, region=fbc323e3b53470e4b5a74c9688484c26, ASSIGN in 215 msec 2023-02-13 17:53:56,695 DEBUG [PEWorker-2] master.DeadServer(114): Removed jenkins-hbase4.apache.org,42905,1676310828981 from processing; numProcessing=0 2023-02-13 17:53:56,697 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=18, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,42905,1676310828981, splitWal=true, meta=true in 1.6950 sec 2023-02-13 17:53:56,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37835] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 17:53:56,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37835] ipc.CallRunner(144): callId: 58 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:46056 deadline: 1676310896903, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 17:53:57,327 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-02-13 17:53:57,442 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 17:53:57,443 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-02-13 17:53:57,466 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-02-13 17:53:57,471 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:53:57,473 INFO [RS-EventLoopGroup-15-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49524, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.8 (auth:SIMPLE), service=ClientService 2023-02-13 17:53:57,487 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-13 17:53:57,491 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-13 17:53:57,491 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.649sec 2023-02-13 17:53:57,491 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-13 17:53:57,492 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-13 17:53:57,492 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-13 17:53:57,492 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37835,1676310834816-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-13 17:53:57,492 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37835,1676310834816-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-13 17:53:57,496 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-13 17:53:57,917 INFO [Listener at localhost/40181] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:57,925 INFO [Listener at localhost/40181] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:57,931 DEBUG [hconnection-0x4190e4f6-shared-pool-6] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 17:53:57,933 INFO [RS-EventLoopGroup-15-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49530, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 17:53:57,934 INFO [Listener at localhost/40181] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 17:53:57,934 INFO [Listener at localhost/40181] master.SnapshotOfRegionAssignmentFromMeta(102): Start to scan the hbase:meta for the current region assignment snappshot 2023-02-13 17:53:57,937 INFO [Listener at localhost/40181] master.SnapshotOfRegionAssignmentFromMeta(164): Finished to scan the hbase:meta for the current region assignmentsnapshot 2023-02-13 17:53:57,937 INFO [Listener at localhost/40181] master.TestRetainAssignmentOnRestart(119): Key={ENCODED => d92fcd787d4162ea4435359fae28fa7d, NAME => 'restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,34875,1676310828951, currentServer=jenkins-hbase4.apache.org,34875,1676310835269 2023-02-13 17:53:57,938 INFO [Listener at localhost/40181] master.TestRetainAssignmentOnRestart(119): Key={ENCODED => 28d66b4066b494c8b8644327cd7fdb67, NAME => 'hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,34875,1676310828951, currentServer=jenkins-hbase4.apache.org,34875,1676310835269 2023-02-13 17:53:57,938 INFO [Listener at localhost/40181] master.TestRetainAssignmentOnRestart(119): Key={ENCODED => fbc323e3b53470e4b5a74c9688484c26, NAME => 'restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,42905,1676310828981, currentServer=jenkins-hbase4.apache.org,42905,1676310835369 2023-02-13 17:53:57,938 INFO [Listener at localhost/40181] master.TestRetainAssignmentOnRestart(119): Key={ENCODED => 52ceb95a47ba04015d52c77bc1874032, NAME => 'restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,42905,1676310828981, currentServer=jenkins-hbase4.apache.org,42905,1676310835369 2023-02-13 17:53:57,938 INFO [Listener at localhost/40181] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-13 17:53:57,938 INFO [Listener at localhost/40181] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-13 17:53:57,938 DEBUG [Listener at localhost/40181] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x410494f3 to 127.0.0.1:53630 2023-02-13 17:53:57,938 DEBUG [Listener at localhost/40181] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:57,939 DEBUG [Listener at localhost/40181] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-13 17:53:57,939 DEBUG [Listener at localhost/40181] util.JVMClusterUtil(257): Found active master hash=576022098, stopped=false 2023-02-13 17:53:57,939 INFO [Listener at localhost/40181] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,37835,1676310834816 2023-02-13 17:53:57,940 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:42905-0x1004ed140fc000f, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 17:53:57,941 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 17:53:57,940 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:39461-0x1004ed140fc000a, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 17:53:57,940 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:34875-0x1004ed140fc000d, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 17:53:57,941 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:53:57,941 INFO [Listener at localhost/40181] procedure2.ProcedureExecutor(629): Stopping 2023-02-13 17:53:57,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42905-0x1004ed140fc000f, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 17:53:57,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39461-0x1004ed140fc000a, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 17:53:57,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 17:53:57,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34875-0x1004ed140fc000d, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 17:53:57,943 DEBUG [Listener at localhost/40181] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x77983962 to 127.0.0.1:53630 2023-02-13 17:53:57,943 DEBUG [Listener at localhost/40181] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:57,944 INFO [Listener at localhost/40181] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,39461,1676310834849' ***** 2023-02-13 17:53:57,944 INFO [Listener at localhost/40181] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 17:53:57,944 INFO [Listener at localhost/40181] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,34875,1676310835269' ***** 2023-02-13 17:53:57,944 INFO [Listener at localhost/40181] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 17:53:57,944 INFO [RS:3;jenkins-hbase4:39461] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 17:53:57,944 INFO [RS:4;jenkins-hbase4:34875] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 17:53:57,944 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 17:53:57,944 INFO [RS:3;jenkins-hbase4:39461] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 17:53:57,944 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 17:53:57,944 INFO [Listener at localhost/40181] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,42905,1676310835369' ***** 2023-02-13 17:53:57,944 INFO [RS:3;jenkins-hbase4:39461] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 17:53:57,944 INFO [RS:4;jenkins-hbase4:34875] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 17:53:57,946 INFO [RS:3;jenkins-hbase4:39461] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,39461,1676310834849 2023-02-13 17:53:57,946 INFO [Listener at localhost/40181] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 17:53:57,946 DEBUG [RS:3;jenkins-hbase4:39461] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0d7c72b1 to 127.0.0.1:53630 2023-02-13 17:53:57,946 INFO [RS:4;jenkins-hbase4:34875] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 17:53:57,946 DEBUG [RS:3;jenkins-hbase4:39461] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:57,946 INFO [RS:3;jenkins-hbase4:39461] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,39461,1676310834849; all regions closed. 2023-02-13 17:53:57,946 INFO [RS:4;jenkins-hbase4:34875] regionserver.HRegionServer(3304): Received CLOSE for d92fcd787d4162ea4435359fae28fa7d 2023-02-13 17:53:57,946 INFO [RS:5;jenkins-hbase4:42905] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 17:53:57,946 INFO [RS:5;jenkins-hbase4:42905] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 17:53:57,946 INFO [RS:5;jenkins-hbase4:42905] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 17:53:57,946 INFO [RS:4;jenkins-hbase4:34875] regionserver.HRegionServer(3304): Received CLOSE for 28d66b4066b494c8b8644327cd7fdb67 2023-02-13 17:53:57,946 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 17:53:57,946 INFO [RS:4;jenkins-hbase4:34875] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,34875,1676310835269 2023-02-13 17:53:57,946 INFO [RS:5;jenkins-hbase4:42905] regionserver.HRegionServer(3304): Received CLOSE for 52ceb95a47ba04015d52c77bc1874032 2023-02-13 17:53:57,948 DEBUG [RS:4;jenkins-hbase4:34875] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x682e4f41 to 127.0.0.1:53630 2023-02-13 17:53:57,948 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(1603): Closing d92fcd787d4162ea4435359fae28fa7d, disabling compactions & flushes 2023-02-13 17:53:57,948 DEBUG [RS:4;jenkins-hbase4:34875] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:57,948 INFO [RS:4;jenkins-hbase4:34875] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-13 17:53:57,948 DEBUG [RS:4;jenkins-hbase4:34875] regionserver.HRegionServer(1479): Online Regions={d92fcd787d4162ea4435359fae28fa7d=restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d., 28d66b4066b494c8b8644327cd7fdb67=hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67.} 2023-02-13 17:53:57,948 DEBUG [RS:4;jenkins-hbase4:34875] regionserver.HRegionServer(1505): Waiting on 28d66b4066b494c8b8644327cd7fdb67, d92fcd787d4162ea4435359fae28fa7d 2023-02-13 17:53:57,948 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(1625): Closing region restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d. 2023-02-13 17:53:57,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1603): Closing 52ceb95a47ba04015d52c77bc1874032, disabling compactions & flushes 2023-02-13 17:53:57,948 INFO [RS:5;jenkins-hbase4:42905] regionserver.HRegionServer(3304): Received CLOSE for fbc323e3b53470e4b5a74c9688484c26 2023-02-13 17:53:57,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d. 2023-02-13 17:53:57,949 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032. 2023-02-13 17:53:57,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(1713): Acquired close lock on restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d. after waiting 0 ms 2023-02-13 17:53:57,949 INFO [RS:5;jenkins-hbase4:42905] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,42905,1676310835369 2023-02-13 17:53:57,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(1723): Updates disabled for region restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d. 2023-02-13 17:53:57,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032. 2023-02-13 17:53:57,949 DEBUG [RS:5;jenkins-hbase4:42905] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2636a12c to 127.0.0.1:53630 2023-02-13 17:53:57,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032. after waiting 0 ms 2023-02-13 17:53:57,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032. 2023-02-13 17:53:57,949 DEBUG [RS:5;jenkins-hbase4:42905] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:57,950 INFO [RS:5;jenkins-hbase4:42905] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 17:53:57,950 INFO [RS:5;jenkins-hbase4:42905] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 17:53:57,950 INFO [RS:5;jenkins-hbase4:42905] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 17:53:57,950 INFO [RS:5;jenkins-hbase4:42905] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-13 17:53:57,960 INFO [RS:5;jenkins-hbase4:42905] regionserver.HRegionServer(1475): Waiting on 3 regions to close 2023-02-13 17:53:57,960 DEBUG [RS:5;jenkins-hbase4:42905] regionserver.HRegionServer(1479): Online Regions={52ceb95a47ba04015d52c77bc1874032=restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032., fbc323e3b53470e4b5a74c9688484c26=restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26., 1588230740=hbase:meta,,1.1588230740} 2023-02-13 17:53:57,961 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 17:53:57,961 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 17:53:57,961 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 17:53:57,961 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 17:53:57,961 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 17:53:57,961 DEBUG [RS:5;jenkins-hbase4:42905] regionserver.HRegionServer(1505): Waiting on 1588230740, 52ceb95a47ba04015d52c77bc1874032, fbc323e3b53470e4b5a74c9688484c26 2023-02-13 17:53:57,961 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=3.81 KB heapSize=6.91 KB 2023-02-13 17:53:57,963 DEBUG [RS:3;jenkins-hbase4:39461] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/oldWALs 2023-02-13 17:53:57,963 INFO [RS:3;jenkins-hbase4:39461] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C39461%2C1676310834849:(num 1676310835372) 2023-02-13 17:53:57,963 DEBUG [RS:3;jenkins-hbase4:39461] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:57,963 INFO [RS:3;jenkins-hbase4:39461] regionserver.LeaseManager(133): Closed leases 2023-02-13 17:53:57,963 INFO [RS:3;jenkins-hbase4:39461] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:39461 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-13 17:53:57,963 INFO [RS:3;jenkins-hbase4:39461] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 17:53:57,963 INFO [RS:3;jenkins-hbase4:39461] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 17:53:57,963 INFO [RS:3;jenkins-hbase4:39461] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 17:53:57,964 INFO [RS:3;jenkins-hbase4:39461] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39461 2023-02-13 17:53:57,965 INFO [regionserver/jenkins-hbase4:39461.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 17:53:57,968 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42905-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableTwo/52ceb95a47ba04015d52c77bc1874032/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-13 17:53:57,968 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:34875-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableThree/d92fcd787d4162ea4435359fae28fa7d/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-13 17:53:57,970 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032. 2023-02-13 17:53:57,970 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1557): Region close journal for 52ceb95a47ba04015d52c77bc1874032: Waiting for close lock at 1676310837948Running coprocessor pre-close hooks at 1676310837948Disabling compacts and flushes for region at 1676310837948Disabling writes for close at 1676310837949 (+1 ms)Writing region close event to WAL at 1676310837961 (+12 ms)Running coprocessor post-close hooks at 1676310837969 (+8 ms)Closed at 1676310837970 (+1 ms) 2023-02-13 17:53:57,970 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42905-0] handler.CloseRegionHandler(117): Closed restartTableTwo,,1676310830843.52ceb95a47ba04015d52c77bc1874032. 2023-02-13 17:53:57,971 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1603): Closing fbc323e3b53470e4b5a74c9688484c26, disabling compactions & flushes 2023-02-13 17:53:57,971 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26. 2023-02-13 17:53:57,971 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26. 2023-02-13 17:53:57,971 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(1837): Closed restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d. 2023-02-13 17:53:57,971 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26. after waiting 0 ms 2023-02-13 17:53:57,971 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(1557): Region close journal for d92fcd787d4162ea4435359fae28fa7d: Waiting for close lock at 1676310837947Running coprocessor pre-close hooks at 1676310837947Disabling compacts and flushes for region at 1676310837947Disabling writes for close at 1676310837949 (+2 ms)Writing region close event to WAL at 1676310837961 (+12 ms)Running coprocessor post-close hooks at 1676310837971 (+10 ms)Closed at 1676310837971 2023-02-13 17:53:57,971 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26. 2023-02-13 17:53:57,971 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:34875-0] handler.CloseRegionHandler(117): Closed restartTableThree,,1676310831459.d92fcd787d4162ea4435359fae28fa7d. 2023-02-13 17:53:57,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(1603): Closing 28d66b4066b494c8b8644327cd7fdb67, disabling compactions & flushes 2023-02-13 17:53:57,973 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67. 2023-02-13 17:53:57,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67. 2023-02-13 17:53:57,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67. after waiting 0 ms 2023-02-13 17:53:57,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67. 2023-02-13 17:53:57,980 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42905-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/default/restartTableOne/fbc323e3b53470e4b5a74c9688484c26/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-13 17:53:57,981 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:34875-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/namespace/28d66b4066b494c8b8644327cd7fdb67/recovered.edits/12.seqid, newMaxSeqId=12, maxSeqId=9 2023-02-13 17:53:57,982 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1837): Closed restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26. 2023-02-13 17:53:57,983 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1557): Region close journal for fbc323e3b53470e4b5a74c9688484c26: Waiting for close lock at 1676310837970Running coprocessor pre-close hooks at 1676310837970Disabling compacts and flushes for region at 1676310837970Disabling writes for close at 1676310837971 (+1 ms)Writing region close event to WAL at 1676310837973 (+2 ms)Running coprocessor post-close hooks at 1676310837981 (+8 ms)Closed at 1676310837982 (+1 ms) 2023-02-13 17:53:57,983 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42905-0] handler.CloseRegionHandler(117): Closed restartTableOne,,1676310830228.fbc323e3b53470e4b5a74c9688484c26. 2023-02-13 17:53:57,984 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67. 2023-02-13 17:53:57,984 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:34875-0] regionserver.HRegion(1557): Region close journal for 28d66b4066b494c8b8644327cd7fdb67: Waiting for close lock at 1676310837972Running coprocessor pre-close hooks at 1676310837972Disabling compacts and flushes for region at 1676310837972Disabling writes for close at 1676310837973 (+1 ms)Writing region close event to WAL at 1676310837976 (+3 ms)Running coprocessor post-close hooks at 1676310837984 (+8 ms)Closed at 1676310837984 2023-02-13 17:53:57,984 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:34875-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676310829732.28d66b4066b494c8b8644327cd7fdb67. 2023-02-13 17:53:57,985 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:42905-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.81 KB at sequenceid=38 (bloomFilter=false), to=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/.tmp/info/dd482650ac5b4d279a0a9999b01c81f9 2023-02-13 17:53:57,992 INFO [regionserver/jenkins-hbase4:42905.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 17:53:57,992 INFO [regionserver/jenkins-hbase4:34875.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 17:53:57,994 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:42905-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/.tmp/info/dd482650ac5b4d279a0a9999b01c81f9 as hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/info/dd482650ac5b4d279a0a9999b01c81f9 2023-02-13 17:53:57,999 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:42905-0] regionserver.HStore(1080): Added hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/info/dd482650ac5b4d279a0a9999b01c81f9, entries=32, sequenceid=38, filesize=8.5 K 2023-02-13 17:53:58,000 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(2947): Finished flush of dataSize ~3.81 KB/3900, heapSize ~6.40 KB/6552, currentSize=0 B/0 for 1588230740 in 39ms, sequenceid=38, compaction requested=false 2023-02-13 17:53:58,009 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:42905-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/data/hbase/meta/1588230740/recovered.edits/41.seqid, newMaxSeqId=41, maxSeqId=27 2023-02-13 17:53:58,010 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:42905-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 17:53:58,010 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 17:53:58,011 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:42905-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676310837960Running coprocessor pre-close hooks at 1676310837960Disabling compacts and flushes for region at 1676310837960Disabling writes for close at 1676310837961 (+1 ms)Obtaining lock to block concurrent updates at 1676310837961Preparing flush snapshotting stores in 1588230740 at 1676310837961Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=3900, getHeapSize=7032, getOffHeapSize=0, getCellsCount=32 at 1676310837961Flushing stores of hbase:meta,,1.1588230740 at 1676310837962 (+1 ms)Flushing 1588230740/info: creating writer at 1676310837963 (+1 ms)Flushing 1588230740/info: appending metadata at 1676310837969 (+6 ms)Flushing 1588230740/info: closing flushed file at 1676310837969Flushing 1588230740/info: reopening flushed file at 1676310837994 (+25 ms)Finished flush of dataSize ~3.81 KB/3900, heapSize ~6.40 KB/6552, currentSize=0 B/0 for 1588230740 in 39ms, sequenceid=38, compaction requested=false at 1676310838000 (+6 ms)Writing region close event to WAL at 1676310838006 (+6 ms)Running coprocessor post-close hooks at 1676310838010 (+4 ms)Closed at 1676310838010 2023-02-13 17:53:58,011 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:42905-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-13 17:53:58,029 INFO [regionserver/jenkins-hbase4:39461.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 17:53:58,052 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:39461-0x1004ed140fc000a, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39461,1676310834849 2023-02-13 17:53:58,052 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:42905-0x1004ed140fc000f, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39461,1676310834849 2023-02-13 17:53:58,052 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:34875-0x1004ed140fc000d, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39461,1676310834849 2023-02-13 17:53:58,052 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:58,052 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:34875-0x1004ed140fc000d, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:58,052 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:42905-0x1004ed140fc000f, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:58,052 ERROR [Listener at localhost/40181-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@68749350 rejected from java.util.concurrent.ThreadPoolExecutor@6c4b0a86[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:58,052 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:39461-0x1004ed140fc000a, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:58,052 ERROR [Listener at localhost/40181-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5408e165 rejected from java.util.concurrent.ThreadPoolExecutor@6c4b0a86[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:58,053 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,39461,1676310834849] 2023-02-13 17:53:58,053 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,39461,1676310834849; numProcessing=1 2023-02-13 17:53:58,054 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,39461,1676310834849 already deleted, retry=false 2023-02-13 17:53:58,054 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,39461,1676310834849 expired; onlineServers=2 2023-02-13 17:53:58,148 INFO [RS:4;jenkins-hbase4:34875] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,34875,1676310835269; all regions closed. 2023-02-13 17:53:58,153 DEBUG [RS:4;jenkins-hbase4:34875] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/oldWALs 2023-02-13 17:53:58,153 INFO [RS:4;jenkins-hbase4:34875] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34875%2C1676310835269:(num 1676310835462) 2023-02-13 17:53:58,153 DEBUG [RS:4;jenkins-hbase4:34875] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:58,154 INFO [RS:4;jenkins-hbase4:34875] regionserver.LeaseManager(133): Closed leases 2023-02-13 17:53:58,154 INFO [RS:4;jenkins-hbase4:34875] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:34875 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 17:53:58,154 INFO [RS:4;jenkins-hbase4:34875] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 17:53:58,154 INFO [regionserver/jenkins-hbase4:34875.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 17:53:58,154 INFO [RS:4;jenkins-hbase4:34875] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 17:53:58,154 INFO [RS:4;jenkins-hbase4:34875] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 17:53:58,155 INFO [RS:4;jenkins-hbase4:34875] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34875 2023-02-13 17:53:58,158 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:58,158 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:34875-0x1004ed140fc000d, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34875,1676310835269 2023-02-13 17:53:58,158 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:42905-0x1004ed140fc000f, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34875,1676310835269 2023-02-13 17:53:58,158 ERROR [Listener at localhost/40181-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4b2dbbc rejected from java.util.concurrent.ThreadPoolExecutor@8d5ff2[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:58,158 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,34875,1676310835269] 2023-02-13 17:53:58,158 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,34875,1676310835269; numProcessing=2 2023-02-13 17:53:58,160 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,34875,1676310835269 already deleted, retry=false 2023-02-13 17:53:58,160 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,34875,1676310835269 expired; onlineServers=1 2023-02-13 17:53:58,161 INFO [RS:5;jenkins-hbase4:42905] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,42905,1676310835369; all regions closed. 2023-02-13 17:53:58,170 DEBUG [RS:5;jenkins-hbase4:42905] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/oldWALs 2023-02-13 17:53:58,170 INFO [RS:5;jenkins-hbase4:42905] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C42905%2C1676310835369.meta:.meta(num 1676310835614) 2023-02-13 17:53:58,180 DEBUG [RS:5;jenkins-hbase4:42905] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/oldWALs 2023-02-13 17:53:58,180 INFO [RS:5;jenkins-hbase4:42905] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C42905%2C1676310835369:(num 1676310835568) 2023-02-13 17:53:58,180 DEBUG [RS:5;jenkins-hbase4:42905] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:58,180 INFO [RS:5;jenkins-hbase4:42905] regionserver.LeaseManager(133): Closed leases 2023-02-13 17:53:58,180 INFO [RS:5;jenkins-hbase4:42905] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:42905 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 17:53:58,181 INFO [regionserver/jenkins-hbase4:42905.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 17:53:58,181 INFO [RS:5;jenkins-hbase4:42905] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42905 2023-02-13 17:53:58,183 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:42905-0x1004ed140fc000f, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42905,1676310835369 2023-02-13 17:53:58,183 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 17:53:58,183 ERROR [Listener at localhost/40181-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2dfe26f7 rejected from java.util.concurrent.ThreadPoolExecutor@29c1a790[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:58,184 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,42905,1676310835369] 2023-02-13 17:53:58,184 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,42905,1676310835369; numProcessing=3 2023-02-13 17:53:58,189 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,42905,1676310835369 already deleted, retry=false 2023-02-13 17:53:58,189 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,42905,1676310835369 expired; onlineServers=0 2023-02-13 17:53:58,189 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,37835,1676310834816' ***** 2023-02-13 17:53:58,189 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-13 17:53:58,190 DEBUG [M:1;jenkins-hbase4:37835] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@266ae6c0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 17:53:58,190 INFO [M:1;jenkins-hbase4:37835] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,37835,1676310834816 2023-02-13 17:53:58,190 INFO [M:1;jenkins-hbase4:37835] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,37835,1676310834816; all regions closed. 2023-02-13 17:53:58,190 DEBUG [M:1;jenkins-hbase4:37835] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 17:53:58,190 DEBUG [M:1;jenkins-hbase4:37835] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-13 17:53:58,190 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-13 17:53:58,190 DEBUG [M:1;jenkins-hbase4:37835] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-13 17:53:58,190 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676310835034] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676310835034,5,FailOnTimeoutGroup] 2023-02-13 17:53:58,190 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676310835034] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676310835034,5,FailOnTimeoutGroup] 2023-02-13 17:53:58,191 INFO [M:1;jenkins-hbase4:37835] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-13 17:53:58,192 INFO [M:1;jenkins-hbase4:37835] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-13 17:53:58,192 INFO [M:1;jenkins-hbase4:37835] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-13 17:53:58,192 DEBUG [M:1;jenkins-hbase4:37835] master.HMaster(1502): Stopping service threads 2023-02-13 17:53:58,192 INFO [M:1;jenkins-hbase4:37835] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-13 17:53:58,192 ERROR [M:1;jenkins-hbase4:37835] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-13 17:53:58,192 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 17:53:58,192 INFO [M:1;jenkins-hbase4:37835] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-13 17:53:58,192 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 17:53:58,193 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-13 17:53:58,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 17:53:58,193 DEBUG [M:1;jenkins-hbase4:37835] zookeeper.ZKUtil(398): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-13 17:53:58,193 WARN [M:1;jenkins-hbase4:37835] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-13 17:53:58,193 INFO [M:1;jenkins-hbase4:37835] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-13 17:53:58,194 INFO [M:1;jenkins-hbase4:37835] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-13 17:53:58,194 DEBUG [M:1;jenkins-hbase4:37835] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 17:53:58,194 INFO [M:1;jenkins-hbase4:37835] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:53:58,194 DEBUG [M:1;jenkins-hbase4:37835] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:53:58,194 DEBUG [M:1;jenkins-hbase4:37835] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 17:53:58,194 DEBUG [M:1;jenkins-hbase4:37835] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:53:58,194 INFO [M:1;jenkins-hbase4:37835] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=38.74 KB heapSize=47.95 KB 2023-02-13 17:53:58,208 INFO [M:1;jenkins-hbase4:37835] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.74 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/79ec8f3ee7e840a1971096c79ca32450 2023-02-13 17:53:58,214 DEBUG [M:1;jenkins-hbase4:37835] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/79ec8f3ee7e840a1971096c79ca32450 as hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/79ec8f3ee7e840a1971096c79ca32450 2023-02-13 17:53:58,219 INFO [M:1;jenkins-hbase4:37835] regionserver.HStore(1080): Added hdfs://localhost:36959/user/jenkins/test-data/8e3ea9ab-8702-dae8-28f0-63f43ed8384e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/79ec8f3ee7e840a1971096c79ca32450, entries=13, sequenceid=246, filesize=6.2 K 2023-02-13 17:53:58,220 INFO [M:1;jenkins-hbase4:37835] regionserver.HRegion(2947): Finished flush of dataSize ~38.74 KB/39674, heapSize ~47.94 KB/49088, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 26ms, sequenceid=246, compaction requested=false 2023-02-13 17:53:58,222 INFO [M:1;jenkins-hbase4:37835] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 17:53:58,222 DEBUG [M:1;jenkins-hbase4:37835] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676310838194Disabling compacts and flushes for region at 1676310838194Disabling writes for close at 1676310838194Obtaining lock to block concurrent updates at 1676310838194Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676310838194Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=39674, getHeapSize=49088, getOffHeapSize=0, getCellsCount=121 at 1676310838194Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676310838195 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676310838196 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676310838199 (+3 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676310838199Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676310838215 (+16 ms)Finished flush of dataSize ~38.74 KB/39674, heapSize ~47.94 KB/49088, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 26ms, sequenceid=246, compaction requested=false at 1676310838220 (+5 ms)Writing region close event to WAL at 1676310838222 (+2 ms)Closed at 1676310838222 2023-02-13 17:53:58,225 INFO [M:1;jenkins-hbase4:37835] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-13 17:53:58,225 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 17:53:58,225 INFO [M:1;jenkins-hbase4:37835] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:37835 2023-02-13 17:53:58,228 DEBUG [M:1;jenkins-hbase4:37835] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,37835,1676310834816 already deleted, retry=false 2023-02-13 17:53:58,342 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:58,342 INFO [M:1;jenkins-hbase4:37835] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,37835,1676310834816; zookeeper connection closed. 2023-02-13 17:53:58,342 ERROR [Listener at localhost/40181-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@619c9e1b rejected from java.util.concurrent.ThreadPoolExecutor@65a306f0[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 20] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:58,342 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): master:37835-0x1004ed140fc0009, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:58,342 ERROR [Listener at localhost/40181-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1422dad9 rejected from java.util.concurrent.ThreadPoolExecutor@65a306f0[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 20] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:58,442 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:42905-0x1004ed140fc000f, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:58,442 INFO [RS:5;jenkins-hbase4:42905] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,42905,1676310835369; zookeeper connection closed. 2023-02-13 17:53:58,442 ERROR [Listener at localhost/40181-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1f0b378 rejected from java.util.concurrent.ThreadPoolExecutor@29c1a790[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:58,442 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@7af9897] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@7af9897 2023-02-13 17:53:58,442 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:42905-0x1004ed140fc000f, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:58,442 ERROR [Listener at localhost/40181-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@793c2524 rejected from java.util.concurrent.ThreadPoolExecutor@29c1a790[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:58,542 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:34875-0x1004ed140fc000d, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:58,542 INFO [RS:4;jenkins-hbase4:34875] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,34875,1676310835269; zookeeper connection closed. 2023-02-13 17:53:58,542 ERROR [Listener at localhost/40181-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@667d2c90 rejected from java.util.concurrent.ThreadPoolExecutor@8d5ff2[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:58,542 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:34875-0x1004ed140fc000d, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:58,542 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@1ed594c] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@1ed594c 2023-02-13 17:53:58,542 ERROR [Listener at localhost/40181-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1e98d775 rejected from java.util.concurrent.ThreadPoolExecutor@8d5ff2[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:58,642 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:39461-0x1004ed140fc000a, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:58,642 INFO [RS:3;jenkins-hbase4:39461] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,39461,1676310834849; zookeeper connection closed. 2023-02-13 17:53:58,642 ERROR [Listener at localhost/40181-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2c01300e rejected from java.util.concurrent.ThreadPoolExecutor@6c4b0a86[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:58,643 DEBUG [Listener at localhost/40181-EventThread] zookeeper.ZKWatcher(600): regionserver:39461-0x1004ed140fc000a, quorum=127.0.0.1:53630, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 17:53:58,643 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5b33c9ba] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5b33c9ba 2023-02-13 17:53:58,643 ERROR [Listener at localhost/40181-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5a76a5a4 rejected from java.util.concurrent.ThreadPoolExecutor@6c4b0a86[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 17:53:58,643 INFO [Listener at localhost/40181] util.JVMClusterUtil(335): Shutdown of 2 master(s) and 6 regionserver(s) complete 2023-02-13 17:53:58,644 WARN [Listener at localhost/40181] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 17:53:58,648 INFO [Listener at localhost/40181] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 17:53:58,752 WARN [BP-2109202673-172.31.14.131-1676310828006 heartbeating to localhost/127.0.0.1:36959] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 17:53:58,752 WARN [BP-2109202673-172.31.14.131-1676310828006 heartbeating to localhost/127.0.0.1:36959] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-2109202673-172.31.14.131-1676310828006 (Datanode Uuid 5b44eaac-9553-466f-8994-1edc7d6b673f) service to localhost/127.0.0.1:36959 2023-02-13 17:53:58,753 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/cluster_5150fa22-ca9c-c15d-e00c-0d4af63a07cb/dfs/data/data5/current/BP-2109202673-172.31.14.131-1676310828006] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 17:53:58,753 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/cluster_5150fa22-ca9c-c15d-e00c-0d4af63a07cb/dfs/data/data6/current/BP-2109202673-172.31.14.131-1676310828006] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 17:53:58,754 WARN [Listener at localhost/40181] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 17:53:58,762 INFO [Listener at localhost/40181] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 17:53:58,867 WARN [BP-2109202673-172.31.14.131-1676310828006 heartbeating to localhost/127.0.0.1:36959] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 17:53:58,867 WARN [BP-2109202673-172.31.14.131-1676310828006 heartbeating to localhost/127.0.0.1:36959] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-2109202673-172.31.14.131-1676310828006 (Datanode Uuid 8d40f6dd-bd2c-4b79-b3dd-37281edb3cc3) service to localhost/127.0.0.1:36959 2023-02-13 17:53:58,868 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/cluster_5150fa22-ca9c-c15d-e00c-0d4af63a07cb/dfs/data/data3/current/BP-2109202673-172.31.14.131-1676310828006] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 17:53:58,868 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/cluster_5150fa22-ca9c-c15d-e00c-0d4af63a07cb/dfs/data/data4/current/BP-2109202673-172.31.14.131-1676310828006] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 17:53:58,869 WARN [Listener at localhost/40181] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 17:53:58,874 INFO [Listener at localhost/40181] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 17:53:58,977 WARN [BP-2109202673-172.31.14.131-1676310828006 heartbeating to localhost/127.0.0.1:36959] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 17:53:58,977 WARN [BP-2109202673-172.31.14.131-1676310828006 heartbeating to localhost/127.0.0.1:36959] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-2109202673-172.31.14.131-1676310828006 (Datanode Uuid fa18d43c-49c0-4428-a5dc-0045e2c3f0c7) service to localhost/127.0.0.1:36959 2023-02-13 17:53:58,977 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/cluster_5150fa22-ca9c-c15d-e00c-0d4af63a07cb/dfs/data/data1/current/BP-2109202673-172.31.14.131-1676310828006] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 17:53:58,978 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d7083a75-d4a4-5136-b18b-0f47226c5a14/cluster_5150fa22-ca9c-c15d-e00c-0d4af63a07cb/dfs/data/data2/current/BP-2109202673-172.31.14.131-1676310828006] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 17:53:58,988 INFO [Listener at localhost/40181] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 17:53:59,104 INFO [Listener at localhost/40181] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-13 17:53:59,135 INFO [Listener at localhost/40181] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-02-13 17:53:59,150 INFO [Listener at localhost/40181] hbase.ResourceChecker(175): after: master.TestRetainAssignmentOnRestartSplitWithoutZk#testRetainAssignmentOnClusterRestart Thread=155 (was 99) Potentially hanging thread: IPC Client (211326415) connection to localhost/127.0.0.1:36959 from jenkins.hfs.7 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-16-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.5@localhost:36959 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.6@localhost:36959 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-12-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-11-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (211326415) connection to localhost/127.0.0.1:36959 from jenkins.hfs.5 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: regionserver/jenkins-hbase4:39461.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: Timer for 'DataNode' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS-EventLoopGroup-10-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-9-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.7@localhost:36959 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost/40181 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.8@localhost:36959 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (211326415) connection to localhost/127.0.0.1:36959 from jenkins.hfs.9 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-15-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: RS-EventLoopGroup-15-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (211326415) connection to localhost/127.0.0.1:36959 from jenkins.hfs.8 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-11-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-13-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-13-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-15-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (211326415) connection to localhost/127.0.0.1:36959 from jenkins.hfs.11 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: LeaseRenewer:jenkins@localhost:36959 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-13-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-12-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-12-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-8-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.10@localhost:36959 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-12-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (211326415) connection to localhost/127.0.0.1:36959 from jenkins.hfs.6 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-12-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (211326415) connection to localhost/127.0.0.1:36959 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-8-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-13-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:34875.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: IPC Client (211326415) connection to localhost/127.0.0.1:36959 from jenkins.hfs.10 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-9-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-16-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-13-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: nioEventLoopGroup-11-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-11-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (211326415) connection to localhost/127.0.0.1:36959 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: LeaseRenewer:jenkins.hfs.11@localhost:36959 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-8-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-12-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-16-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.9@localhost:36959 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-13-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:42905.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-9-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=584 (was 504) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=336 (was 379), ProcessCount=168 (was 168), AvailableMemoryMB=1110 (was 1399)