2023-07-12 20:08:19,957 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestScannerTimeout timeout: 13 mins 2023-07-12 20:08:20,175 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6 2023-07-12 20:08:20,188 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=2, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-12 20:08:20,188 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/cluster_b448c56c-3a9d-1fce-b944-8d379dcf0ce4, deleteOnExit=true 2023-07-12 20:08:20,189 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-12 20:08:20,189 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/test.cache.data in system properties and HBase conf 2023-07-12 20:08:20,189 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/hadoop.tmp.dir in system properties and HBase conf 2023-07-12 20:08:20,190 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/hadoop.log.dir in system properties and HBase conf 2023-07-12 20:08:20,191 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-12 20:08:20,191 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-12 20:08:20,191 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-12 20:08:20,309 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-12 20:08:20,705 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-12 20:08:20,709 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-12 20:08:20,709 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-12 20:08:20,710 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-12 20:08:20,710 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 20:08:20,710 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-12 20:08:20,710 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-12 20:08:20,711 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 20:08:20,711 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 20:08:20,711 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-12 20:08:20,712 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/nfs.dump.dir in system properties and HBase conf 2023-07-12 20:08:20,712 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/java.io.tmpdir in system properties and HBase conf 2023-07-12 20:08:20,713 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 20:08:20,713 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-12 20:08:20,713 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-12 20:08:21,258 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 20:08:21,262 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 20:08:21,548 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-12 20:08:21,711 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-12 20:08:21,727 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 20:08:21,765 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 20:08:21,797 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/java.io.tmpdir/Jetty_localhost_36607_hdfs____ttdxd0/webapp 2023-07-12 20:08:21,951 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36607 2023-07-12 20:08:21,961 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 20:08:21,961 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 20:08:22,341 WARN [Listener at localhost/45739] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 20:08:22,453 WARN [Listener at localhost/45739] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-12 20:08:22,470 WARN [Listener at localhost/45739] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 20:08:22,476 INFO [Listener at localhost/45739] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 20:08:22,480 INFO [Listener at localhost/45739] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/java.io.tmpdir/Jetty_localhost_46299_datanode____zhhed3/webapp 2023-07-12 20:08:22,575 INFO [Listener at localhost/45739] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46299 2023-07-12 20:08:22,892 WARN [Listener at localhost/35153] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 20:08:22,905 WARN [Listener at localhost/35153] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-12 20:08:22,909 WARN [Listener at localhost/35153] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 20:08:22,911 INFO [Listener at localhost/35153] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 20:08:22,920 INFO [Listener at localhost/35153] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/java.io.tmpdir/Jetty_localhost_34353_datanode____.okwcuh/webapp 2023-07-12 20:08:23,029 INFO [Listener at localhost/35153] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34353 2023-07-12 20:08:23,038 WARN [Listener at localhost/44421] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 20:08:23,355 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x48932c977ed772ed: Processing first storage report for DS-8370ea8b-9cf9-4e9e-a7c3-2af949363208 from datanode 5ec7b338-4711-43ff-baaf-745d20ec4762 2023-07-12 20:08:23,357 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x48932c977ed772ed: from storage DS-8370ea8b-9cf9-4e9e-a7c3-2af949363208 node DatanodeRegistration(127.0.0.1:41651, datanodeUuid=5ec7b338-4711-43ff-baaf-745d20ec4762, infoPort=46145, infoSecurePort=0, ipcPort=35153, storageInfo=lv=-57;cid=testClusterID;nsid=1363288808;c=1689192501335), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-12 20:08:23,357 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc7b299c1852166ce: Processing first storage report for DS-c995922d-4442-40c9-9e1a-e075f395e478 from datanode 266107c9-b366-44ad-b66f-f6bdb7b07e21 2023-07-12 20:08:23,357 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc7b299c1852166ce: from storage DS-c995922d-4442-40c9-9e1a-e075f395e478 node DatanodeRegistration(127.0.0.1:43453, datanodeUuid=266107c9-b366-44ad-b66f-f6bdb7b07e21, infoPort=37201, infoSecurePort=0, ipcPort=44421, storageInfo=lv=-57;cid=testClusterID;nsid=1363288808;c=1689192501335), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 20:08:23,357 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x48932c977ed772ed: Processing first storage report for DS-5387efc9-4302-4e6e-907c-85284af97cef from datanode 5ec7b338-4711-43ff-baaf-745d20ec4762 2023-07-12 20:08:23,358 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x48932c977ed772ed: from storage DS-5387efc9-4302-4e6e-907c-85284af97cef node DatanodeRegistration(127.0.0.1:41651, datanodeUuid=5ec7b338-4711-43ff-baaf-745d20ec4762, infoPort=46145, infoSecurePort=0, ipcPort=35153, storageInfo=lv=-57;cid=testClusterID;nsid=1363288808;c=1689192501335), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 20:08:23,358 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc7b299c1852166ce: Processing first storage report for DS-a2666000-7940-4db8-bded-a65bbeac483d from datanode 266107c9-b366-44ad-b66f-f6bdb7b07e21 2023-07-12 20:08:23,358 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc7b299c1852166ce: from storage DS-a2666000-7940-4db8-bded-a65bbeac483d node DatanodeRegistration(127.0.0.1:43453, datanodeUuid=266107c9-b366-44ad-b66f-f6bdb7b07e21, infoPort=37201, infoSecurePort=0, ipcPort=44421, storageInfo=lv=-57;cid=testClusterID;nsid=1363288808;c=1689192501335), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 20:08:23,427 DEBUG [Listener at localhost/44421] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6 2023-07-12 20:08:23,488 INFO [Listener at localhost/44421] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/cluster_b448c56c-3a9d-1fce-b944-8d379dcf0ce4/zookeeper_0, clientPort=50668, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/cluster_b448c56c-3a9d-1fce-b944-8d379dcf0ce4/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/cluster_b448c56c-3a9d-1fce-b944-8d379dcf0ce4/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-12 20:08:23,500 INFO [Listener at localhost/44421] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=50668 2023-07-12 20:08:23,508 INFO [Listener at localhost/44421] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 20:08:23,510 INFO [Listener at localhost/44421] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 20:08:24,160 INFO [Listener at localhost/44421] util.FSUtils(471): Created version file at hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4 with version=8 2023-07-12 20:08:24,161 INFO [Listener at localhost/44421] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/hbase-staging 2023-07-12 20:08:24,494 INFO [Listener at localhost/44421] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-12 20:08:24,960 INFO [Listener at localhost/44421] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-12 20:08:24,991 INFO [Listener at localhost/44421] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 20:08:24,992 INFO [Listener at localhost/44421] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 20:08:24,992 INFO [Listener at localhost/44421] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 20:08:24,992 INFO [Listener at localhost/44421] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 20:08:24,992 INFO [Listener at localhost/44421] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 20:08:25,132 INFO [Listener at localhost/44421] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 20:08:25,218 DEBUG [Listener at localhost/44421] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-12 20:08:25,311 INFO [Listener at localhost/44421] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34965 2023-07-12 20:08:25,320 INFO [Listener at localhost/44421] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 20:08:25,322 INFO [Listener at localhost/44421] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 20:08:25,343 INFO [Listener at localhost/44421] zookeeper.RecoverableZooKeeper(93): Process identifier=master:34965 connecting to ZooKeeper ensemble=127.0.0.1:50668 2023-07-12 20:08:25,383 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:349650x0, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 20:08:25,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:34965-0x1015b269bb50000 connected 2023-07-12 20:08:25,411 DEBUG [Listener at localhost/44421] zookeeper.ZKUtil(164): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 20:08:25,412 DEBUG [Listener at localhost/44421] zookeeper.ZKUtil(164): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 20:08:25,415 DEBUG [Listener at localhost/44421] zookeeper.ZKUtil(164): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 20:08:25,423 DEBUG [Listener at localhost/44421] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34965 2023-07-12 20:08:25,423 DEBUG [Listener at localhost/44421] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34965 2023-07-12 20:08:25,423 DEBUG [Listener at localhost/44421] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34965 2023-07-12 20:08:25,424 DEBUG [Listener at localhost/44421] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34965 2023-07-12 20:08:25,426 DEBUG [Listener at localhost/44421] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34965 2023-07-12 20:08:25,432 INFO [Listener at localhost/44421] master.HMaster(444): hbase.rootdir=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4, hbase.cluster.distributed=false 2023-07-12 20:08:25,497 INFO [Listener at localhost/44421] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-12 20:08:25,498 INFO [Listener at localhost/44421] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 20:08:25,498 INFO [Listener at localhost/44421] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 20:08:25,498 INFO [Listener at localhost/44421] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 20:08:25,498 INFO [Listener at localhost/44421] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 20:08:25,498 INFO [Listener at localhost/44421] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 20:08:25,503 INFO [Listener at localhost/44421] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 20:08:25,506 INFO [Listener at localhost/44421] ipc.NettyRpcServer(120): Bind to /172.31.14.131:43253 2023-07-12 20:08:25,508 INFO [Listener at localhost/44421] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 20:08:25,514 DEBUG [Listener at localhost/44421] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 20:08:25,515 INFO [Listener at localhost/44421] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 20:08:25,517 INFO [Listener at localhost/44421] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 20:08:25,518 INFO [Listener at localhost/44421] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:43253 connecting to ZooKeeper ensemble=127.0.0.1:50668 2023-07-12 20:08:25,522 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): regionserver:432530x0, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 20:08:25,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:43253-0x1015b269bb50001 connected 2023-07-12 20:08:25,523 DEBUG [Listener at localhost/44421] zookeeper.ZKUtil(164): regionserver:43253-0x1015b269bb50001, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 20:08:25,524 DEBUG [Listener at localhost/44421] zookeeper.ZKUtil(164): regionserver:43253-0x1015b269bb50001, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 20:08:25,525 DEBUG [Listener at localhost/44421] zookeeper.ZKUtil(164): regionserver:43253-0x1015b269bb50001, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 20:08:25,525 DEBUG [Listener at localhost/44421] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43253 2023-07-12 20:08:25,525 DEBUG [Listener at localhost/44421] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43253 2023-07-12 20:08:25,526 DEBUG [Listener at localhost/44421] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43253 2023-07-12 20:08:25,527 DEBUG [Listener at localhost/44421] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43253 2023-07-12 20:08:25,527 DEBUG [Listener at localhost/44421] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43253 2023-07-12 20:08:25,541 INFO [Listener at localhost/44421] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-12 20:08:25,541 INFO [Listener at localhost/44421] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 20:08:25,541 INFO [Listener at localhost/44421] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 20:08:25,542 INFO [Listener at localhost/44421] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 20:08:25,542 INFO [Listener at localhost/44421] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 20:08:25,542 INFO [Listener at localhost/44421] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 20:08:25,542 INFO [Listener at localhost/44421] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 20:08:25,544 INFO [Listener at localhost/44421] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42213 2023-07-12 20:08:25,544 INFO [Listener at localhost/44421] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 20:08:25,546 DEBUG [Listener at localhost/44421] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 20:08:25,547 INFO [Listener at localhost/44421] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 20:08:25,550 INFO [Listener at localhost/44421] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 20:08:25,552 INFO [Listener at localhost/44421] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:42213 connecting to ZooKeeper ensemble=127.0.0.1:50668 2023-07-12 20:08:25,558 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): regionserver:422130x0, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 20:08:25,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:42213-0x1015b269bb50002 connected 2023-07-12 20:08:25,559 DEBUG [Listener at localhost/44421] zookeeper.ZKUtil(164): regionserver:42213-0x1015b269bb50002, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 20:08:25,560 DEBUG [Listener at localhost/44421] zookeeper.ZKUtil(164): regionserver:42213-0x1015b269bb50002, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 20:08:25,561 DEBUG [Listener at localhost/44421] zookeeper.ZKUtil(164): regionserver:42213-0x1015b269bb50002, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 20:08:25,561 DEBUG [Listener at localhost/44421] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42213 2023-07-12 20:08:25,562 DEBUG [Listener at localhost/44421] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42213 2023-07-12 20:08:25,562 DEBUG [Listener at localhost/44421] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42213 2023-07-12 20:08:25,562 DEBUG [Listener at localhost/44421] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42213 2023-07-12 20:08:25,564 DEBUG [Listener at localhost/44421] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42213 2023-07-12 20:08:25,565 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,34965,1689192504325 2023-07-12 20:08:25,573 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 20:08:25,575 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,34965,1689192504325 2023-07-12 20:08:25,594 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 20:08:25,594 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): regionserver:43253-0x1015b269bb50001, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 20:08:25,594 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): regionserver:42213-0x1015b269bb50002, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 20:08:25,594 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 20:08:25,596 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 20:08:25,597 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,34965,1689192504325 from backup master directory 2023-07-12 20:08:25,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 20:08:25,600 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,34965,1689192504325 2023-07-12 20:08:25,600 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 20:08:25,601 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 20:08:25,601 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,34965,1689192504325 2023-07-12 20:08:25,603 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-12 20:08:25,605 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-12 20:08:25,687 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/hbase.id with ID: 0fd096bf-885b-42ee-9942-e5df6a803c4e 2023-07-12 20:08:25,728 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 20:08:25,743 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 20:08:25,785 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x16d66d79 to 127.0.0.1:50668 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 20:08:25,818 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2bc60790, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 20:08:25,840 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 20:08:25,842 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 20:08:25,858 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-12 20:08:25,858 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-12 20:08:25,860 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-12 20:08:25,864 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-12 20:08:25,865 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 20:08:25,897 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/MasterData/data/master/store-tmp 2023-07-12 20:08:25,927 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 20:08:25,927 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 20:08:25,927 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 20:08:25,927 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 20:08:25,927 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 20:08:25,927 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 20:08:25,927 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 20:08:25,928 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 20:08:25,929 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/MasterData/WALs/jenkins-hbase4.apache.org,34965,1689192504325 2023-07-12 20:08:25,948 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34965%2C1689192504325, suffix=, logDir=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/MasterData/WALs/jenkins-hbase4.apache.org,34965,1689192504325, archiveDir=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/MasterData/oldWALs, maxLogs=10 2023-07-12 20:08:26,006 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41651,DS-8370ea8b-9cf9-4e9e-a7c3-2af949363208,DISK] 2023-07-12 20:08:26,006 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43453,DS-c995922d-4442-40c9-9e1a-e075f395e478,DISK] 2023-07-12 20:08:26,014 DEBUG [RS-EventLoopGroup-4-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 20:08:26,080 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/MasterData/WALs/jenkins-hbase4.apache.org,34965,1689192504325/jenkins-hbase4.apache.org%2C34965%2C1689192504325.1689192505958 2023-07-12 20:08:26,082 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43453,DS-c995922d-4442-40c9-9e1a-e075f395e478,DISK], DatanodeInfoWithStorage[127.0.0.1:41651,DS-8370ea8b-9cf9-4e9e-a7c3-2af949363208,DISK]] 2023-07-12 20:08:26,082 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 20:08:26,083 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 20:08:26,085 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 20:08:26,087 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 20:08:26,141 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 20:08:26,149 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 20:08:26,179 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 20:08:26,190 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 20:08:26,196 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 20:08:26,198 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 20:08:26,212 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 20:08:26,216 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 20:08:26,218 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69897043, jitterRate=0.041547104716300964}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 20:08:26,218 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 20:08:26,222 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 20:08:26,248 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-12 20:08:26,248 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-12 20:08:26,250 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 20:08:26,252 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-12 20:08:26,284 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-07-12 20:08:26,284 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 20:08:26,308 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-12 20:08:26,314 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-12 20:08:26,345 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 20:08:26,350 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 20:08:26,352 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 20:08:26,358 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 20:08:26,363 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 20:08:26,367 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 20:08:26,369 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 20:08:26,369 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 20:08:26,380 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 20:08:26,385 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): regionserver:43253-0x1015b269bb50001, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 20:08:26,385 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 20:08:26,385 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): regionserver:42213-0x1015b269bb50002, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 20:08:26,385 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 20:08:26,385 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,34965,1689192504325, sessionid=0x1015b269bb50000, setting cluster-up flag (Was=false) 2023-07-12 20:08:26,400 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 20:08:26,406 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 20:08:26,408 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,34965,1689192504325 2023-07-12 20:08:26,413 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 20:08:26,418 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 20:08:26,419 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,34965,1689192504325 2023-07-12 20:08:26,422 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/.hbase-snapshot/.tmp 2023-07-12 20:08:26,467 INFO [RS:1;jenkins-hbase4:42213] regionserver.HRegionServer(951): ClusterId : 0fd096bf-885b-42ee-9942-e5df6a803c4e 2023-07-12 20:08:26,467 INFO [RS:0;jenkins-hbase4:43253] regionserver.HRegionServer(951): ClusterId : 0fd096bf-885b-42ee-9942-e5df6a803c4e 2023-07-12 20:08:26,473 DEBUG [RS:1;jenkins-hbase4:42213] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 20:08:26,473 DEBUG [RS:0;jenkins-hbase4:43253] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 20:08:26,480 DEBUG [RS:1;jenkins-hbase4:42213] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 20:08:26,480 DEBUG [RS:0;jenkins-hbase4:43253] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 20:08:26,480 DEBUG [RS:1;jenkins-hbase4:42213] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 20:08:26,480 DEBUG [RS:0;jenkins-hbase4:43253] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 20:08:26,483 DEBUG [RS:0;jenkins-hbase4:43253] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 20:08:26,483 DEBUG [RS:1;jenkins-hbase4:42213] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 20:08:26,485 DEBUG [RS:0;jenkins-hbase4:43253] zookeeper.ReadOnlyZKClient(139): Connect 0x5c6607c3 to 127.0.0.1:50668 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 20:08:26,486 DEBUG [RS:1;jenkins-hbase4:42213] zookeeper.ReadOnlyZKClient(139): Connect 0x168a8e12 to 127.0.0.1:50668 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 20:08:26,497 DEBUG [RS:1;jenkins-hbase4:42213] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@184d62b3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 20:08:26,497 DEBUG [RS:0;jenkins-hbase4:43253] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52a1f23f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 20:08:26,498 DEBUG [RS:1;jenkins-hbase4:42213] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1c72e562, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-12 20:08:26,498 DEBUG [RS:0;jenkins-hbase4:43253] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@569e6cb3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-12 20:08:26,520 DEBUG [RS:0;jenkins-hbase4:43253] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:43253 2023-07-12 20:08:26,520 DEBUG [RS:1;jenkins-hbase4:42213] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:42213 2023-07-12 20:08:26,524 INFO [RS:1;jenkins-hbase4:42213] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 20:08:26,524 INFO [RS:0;jenkins-hbase4:43253] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 20:08:26,524 INFO [RS:1;jenkins-hbase4:42213] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 20:08:26,524 INFO [RS:0;jenkins-hbase4:43253] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 20:08:26,524 DEBUG [RS:1;jenkins-hbase4:42213] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 20:08:26,524 DEBUG [RS:0;jenkins-hbase4:43253] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 20:08:26,527 INFO [RS:1;jenkins-hbase4:42213] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,34965,1689192504325 with isa=jenkins-hbase4.apache.org/172.31.14.131:42213, startcode=1689192505540 2023-07-12 20:08:26,527 INFO [RS:0;jenkins-hbase4:43253] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,34965,1689192504325 with isa=jenkins-hbase4.apache.org/172.31.14.131:43253, startcode=1689192505497 2023-07-12 20:08:26,545 DEBUG [RS:0;jenkins-hbase4:43253] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 20:08:26,545 DEBUG [RS:1;jenkins-hbase4:42213] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 20:08:26,546 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-12 20:08:26,556 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 20:08:26,557 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 20:08:26,557 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 20:08:26,557 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 20:08:26,557 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-12 20:08:26,557 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:26,557 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-12 20:08:26,558 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:26,564 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689192536564 2023-07-12 20:08:26,567 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 20:08:26,580 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 20:08:26,587 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 20:08:26,588 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 20:08:26,588 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 20:08:26,588 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 20:08:26,589 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:26,591 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 20:08:26,591 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 20:08:26,592 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-12 20:08:26,593 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 20:08:26,593 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 20:08:26,597 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 20:08:26,597 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 20:08:26,598 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 20:08:26,599 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689192506599,5,FailOnTimeoutGroup] 2023-07-12 20:08:26,600 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689192506600,5,FailOnTimeoutGroup] 2023-07-12 20:08:26,600 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:26,600 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 20:08:26,602 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:26,602 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:26,608 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46677, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 20:08:26,608 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48777, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 20:08:26,620 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=34965] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,42213,1689192505540 2023-07-12 20:08:26,634 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 20:08:26,636 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=34965] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,43253,1689192505497 2023-07-12 20:08:26,636 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 20:08:26,636 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4 2023-07-12 20:08:26,641 DEBUG [RS:1;jenkins-hbase4:42213] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4 2023-07-12 20:08:26,641 DEBUG [RS:0;jenkins-hbase4:43253] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4 2023-07-12 20:08:26,641 DEBUG [RS:1;jenkins-hbase4:42213] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:45739 2023-07-12 20:08:26,642 DEBUG [RS:0;jenkins-hbase4:43253] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:45739 2023-07-12 20:08:26,642 DEBUG [RS:1;jenkins-hbase4:42213] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 20:08:26,642 DEBUG [RS:0;jenkins-hbase4:43253] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 20:08:26,649 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 20:08:26,654 DEBUG [RS:1;jenkins-hbase4:42213] zookeeper.ZKUtil(162): regionserver:42213-0x1015b269bb50002, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42213,1689192505540 2023-07-12 20:08:26,654 WARN [RS:1;jenkins-hbase4:42213] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 20:08:26,655 INFO [RS:1;jenkins-hbase4:42213] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 20:08:26,655 DEBUG [RS:1;jenkins-hbase4:42213] regionserver.HRegionServer(1948): logDir=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,42213,1689192505540 2023-07-12 20:08:26,656 DEBUG [RS:0;jenkins-hbase4:43253] zookeeper.ZKUtil(162): regionserver:43253-0x1015b269bb50001, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43253,1689192505497 2023-07-12 20:08:26,656 WARN [RS:0;jenkins-hbase4:43253] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 20:08:26,656 INFO [RS:0;jenkins-hbase4:43253] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 20:08:26,656 DEBUG [RS:0;jenkins-hbase4:43253] regionserver.HRegionServer(1948): logDir=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497 2023-07-12 20:08:26,656 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,42213,1689192505540] 2023-07-12 20:08:26,657 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,43253,1689192505497] 2023-07-12 20:08:26,659 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 20:08:26,664 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 20:08:26,666 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/info 2023-07-12 20:08:26,667 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 20:08:26,668 DEBUG [RS:1;jenkins-hbase4:42213] zookeeper.ZKUtil(162): regionserver:42213-0x1015b269bb50002, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42213,1689192505540 2023-07-12 20:08:26,668 DEBUG [RS:0;jenkins-hbase4:43253] zookeeper.ZKUtil(162): regionserver:43253-0x1015b269bb50001, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42213,1689192505540 2023-07-12 20:08:26,669 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 20:08:26,669 DEBUG [RS:1;jenkins-hbase4:42213] zookeeper.ZKUtil(162): regionserver:42213-0x1015b269bb50002, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43253,1689192505497 2023-07-12 20:08:26,669 DEBUG [RS:0;jenkins-hbase4:43253] zookeeper.ZKUtil(162): regionserver:43253-0x1015b269bb50001, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43253,1689192505497 2023-07-12 20:08:26,669 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 20:08:26,672 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/rep_barrier 2023-07-12 20:08:26,673 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 20:08:26,674 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 20:08:26,674 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 20:08:26,676 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/table 2023-07-12 20:08:26,677 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 20:08:26,678 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 20:08:26,679 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740 2023-07-12 20:08:26,680 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740 2023-07-12 20:08:26,681 DEBUG [RS:1;jenkins-hbase4:42213] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 20:08:26,681 DEBUG [RS:0;jenkins-hbase4:43253] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 20:08:26,686 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 20:08:26,688 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 20:08:26,692 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 20:08:26,692 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60834154, jitterRate=-0.09350046515464783}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 20:08:26,693 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 20:08:26,693 INFO [RS:1;jenkins-hbase4:42213] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 20:08:26,693 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 20:08:26,693 INFO [RS:0;jenkins-hbase4:43253] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 20:08:26,693 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 20:08:26,693 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 20:08:26,693 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 20:08:26,693 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 20:08:26,694 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 20:08:26,695 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 20:08:26,700 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 20:08:26,700 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-12 20:08:26,710 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-12 20:08:26,716 INFO [RS:1;jenkins-hbase4:42213] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 20:08:26,716 INFO [RS:0;jenkins-hbase4:43253] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 20:08:26,720 INFO [RS:1;jenkins-hbase4:42213] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 20:08:26,720 INFO [RS:0;jenkins-hbase4:43253] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 20:08:26,720 INFO [RS:1;jenkins-hbase4:42213] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:26,720 INFO [RS:0;jenkins-hbase4:43253] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:26,721 INFO [RS:0;jenkins-hbase4:43253] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 20:08:26,721 INFO [RS:1;jenkins-hbase4:42213] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 20:08:26,724 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-12 20:08:26,726 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-12 20:08:26,730 INFO [RS:1;jenkins-hbase4:42213] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:26,730 INFO [RS:0;jenkins-hbase4:43253] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:26,730 DEBUG [RS:1;jenkins-hbase4:42213] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:26,731 DEBUG [RS:0;jenkins-hbase4:43253] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:26,731 DEBUG [RS:1;jenkins-hbase4:42213] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:26,731 DEBUG [RS:0;jenkins-hbase4:43253] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:26,731 DEBUG [RS:1;jenkins-hbase4:42213] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:26,731 DEBUG [RS:0;jenkins-hbase4:43253] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:26,731 DEBUG [RS:1;jenkins-hbase4:42213] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:26,731 DEBUG [RS:0;jenkins-hbase4:43253] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:26,731 DEBUG [RS:1;jenkins-hbase4:42213] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:26,731 DEBUG [RS:0;jenkins-hbase4:43253] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:26,731 DEBUG [RS:1;jenkins-hbase4:42213] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-12 20:08:26,731 DEBUG [RS:0;jenkins-hbase4:43253] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-12 20:08:26,732 DEBUG [RS:1;jenkins-hbase4:42213] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:26,732 DEBUG [RS:0;jenkins-hbase4:43253] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:26,732 DEBUG [RS:1;jenkins-hbase4:42213] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:26,732 DEBUG [RS:0;jenkins-hbase4:43253] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:26,732 DEBUG [RS:1;jenkins-hbase4:42213] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:26,732 DEBUG [RS:0;jenkins-hbase4:43253] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:26,732 DEBUG [RS:1;jenkins-hbase4:42213] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:26,732 DEBUG [RS:0;jenkins-hbase4:43253] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:26,734 INFO [RS:0;jenkins-hbase4:43253] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:26,734 INFO [RS:0;jenkins-hbase4:43253] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:26,734 INFO [RS:0;jenkins-hbase4:43253] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:26,734 INFO [RS:1;jenkins-hbase4:42213] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:26,734 INFO [RS:1;jenkins-hbase4:42213] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:26,734 INFO [RS:1;jenkins-hbase4:42213] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:26,750 INFO [RS:1;jenkins-hbase4:42213] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 20:08:26,750 INFO [RS:0;jenkins-hbase4:43253] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 20:08:26,753 INFO [RS:1;jenkins-hbase4:42213] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42213,1689192505540-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:26,753 INFO [RS:0;jenkins-hbase4:43253] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43253,1689192505497-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:26,770 INFO [RS:0;jenkins-hbase4:43253] regionserver.Replication(203): jenkins-hbase4.apache.org,43253,1689192505497 started 2023-07-12 20:08:26,770 INFO [RS:1;jenkins-hbase4:42213] regionserver.Replication(203): jenkins-hbase4.apache.org,42213,1689192505540 started 2023-07-12 20:08:26,770 INFO [RS:0;jenkins-hbase4:43253] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,43253,1689192505497, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:43253, sessionid=0x1015b269bb50001 2023-07-12 20:08:26,771 INFO [RS:1;jenkins-hbase4:42213] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,42213,1689192505540, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:42213, sessionid=0x1015b269bb50002 2023-07-12 20:08:26,771 DEBUG [RS:1;jenkins-hbase4:42213] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 20:08:26,771 DEBUG [RS:0;jenkins-hbase4:43253] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 20:08:26,771 DEBUG [RS:1;jenkins-hbase4:42213] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,42213,1689192505540 2023-07-12 20:08:26,771 DEBUG [RS:0;jenkins-hbase4:43253] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,43253,1689192505497 2023-07-12 20:08:26,772 DEBUG [RS:1;jenkins-hbase4:42213] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42213,1689192505540' 2023-07-12 20:08:26,772 DEBUG [RS:0;jenkins-hbase4:43253] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43253,1689192505497' 2023-07-12 20:08:26,772 DEBUG [RS:1;jenkins-hbase4:42213] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 20:08:26,772 DEBUG [RS:0;jenkins-hbase4:43253] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 20:08:26,773 DEBUG [RS:0;jenkins-hbase4:43253] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 20:08:26,773 DEBUG [RS:1;jenkins-hbase4:42213] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 20:08:26,773 DEBUG [RS:0;jenkins-hbase4:43253] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 20:08:26,773 DEBUG [RS:0;jenkins-hbase4:43253] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 20:08:26,773 DEBUG [RS:1;jenkins-hbase4:42213] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 20:08:26,773 DEBUG [RS:0;jenkins-hbase4:43253] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,43253,1689192505497 2023-07-12 20:08:26,773 DEBUG [RS:1;jenkins-hbase4:42213] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 20:08:26,773 DEBUG [RS:0;jenkins-hbase4:43253] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43253,1689192505497' 2023-07-12 20:08:26,774 DEBUG [RS:1;jenkins-hbase4:42213] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,42213,1689192505540 2023-07-12 20:08:26,774 DEBUG [RS:0;jenkins-hbase4:43253] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 20:08:26,774 DEBUG [RS:1;jenkins-hbase4:42213] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42213,1689192505540' 2023-07-12 20:08:26,774 DEBUG [RS:1;jenkins-hbase4:42213] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 20:08:26,774 DEBUG [RS:0;jenkins-hbase4:43253] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 20:08:26,775 DEBUG [RS:1;jenkins-hbase4:42213] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 20:08:26,775 DEBUG [RS:0;jenkins-hbase4:43253] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 20:08:26,775 DEBUG [RS:1;jenkins-hbase4:42213] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 20:08:26,775 INFO [RS:0;jenkins-hbase4:43253] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 20:08:26,775 INFO [RS:1;jenkins-hbase4:42213] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 20:08:26,775 INFO [RS:0;jenkins-hbase4:43253] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 20:08:26,776 INFO [RS:1;jenkins-hbase4:42213] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 20:08:26,879 DEBUG [jenkins-hbase4:34965] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-12 20:08:26,884 DEBUG [jenkins-hbase4:34965] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-12 20:08:26,889 INFO [RS:1;jenkins-hbase4:42213] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42213%2C1689192505540, suffix=, logDir=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,42213,1689192505540, archiveDir=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/oldWALs, maxLogs=32 2023-07-12 20:08:26,890 INFO [RS:0;jenkins-hbase4:43253] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43253%2C1689192505497, suffix=, logDir=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497, archiveDir=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/oldWALs, maxLogs=32 2023-07-12 20:08:26,891 DEBUG [jenkins-hbase4:34965] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 20:08:26,891 DEBUG [jenkins-hbase4:34965] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 20:08:26,891 DEBUG [jenkins-hbase4:34965] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 20:08:26,894 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,43253,1689192505497, state=OPENING 2023-07-12 20:08:26,901 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-12 20:08:26,903 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 20:08:26,903 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 20:08:26,908 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,43253,1689192505497}] 2023-07-12 20:08:26,914 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43453,DS-c995922d-4442-40c9-9e1a-e075f395e478,DISK] 2023-07-12 20:08:26,914 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41651,DS-8370ea8b-9cf9-4e9e-a7c3-2af949363208,DISK] 2023-07-12 20:08:26,918 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41651,DS-8370ea8b-9cf9-4e9e-a7c3-2af949363208,DISK] 2023-07-12 20:08:26,918 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43453,DS-c995922d-4442-40c9-9e1a-e075f395e478,DISK] 2023-07-12 20:08:26,924 INFO [RS:1;jenkins-hbase4:42213] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,42213,1689192505540/jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894 2023-07-12 20:08:26,924 DEBUG [RS:1;jenkins-hbase4:42213] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43453,DS-c995922d-4442-40c9-9e1a-e075f395e478,DISK], DatanodeInfoWithStorage[127.0.0.1:41651,DS-8370ea8b-9cf9-4e9e-a7c3-2af949363208,DISK]] 2023-07-12 20:08:26,927 INFO [RS:0;jenkins-hbase4:43253] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497/jenkins-hbase4.apache.org%2C43253%2C1689192505497.1689192506894 2023-07-12 20:08:26,927 DEBUG [RS:0;jenkins-hbase4:43253] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43453,DS-c995922d-4442-40c9-9e1a-e075f395e478,DISK], DatanodeInfoWithStorage[127.0.0.1:41651,DS-8370ea8b-9cf9-4e9e-a7c3-2af949363208,DISK]] 2023-07-12 20:08:27,101 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,43253,1689192505497 2023-07-12 20:08:27,103 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 20:08:27,107 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36946, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 20:08:27,119 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-12 20:08:27,119 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 20:08:27,123 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43253%2C1689192505497.meta, suffix=.meta, logDir=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497, archiveDir=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/oldWALs, maxLogs=32 2023-07-12 20:08:27,140 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43453,DS-c995922d-4442-40c9-9e1a-e075f395e478,DISK] 2023-07-12 20:08:27,141 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41651,DS-8370ea8b-9cf9-4e9e-a7c3-2af949363208,DISK] 2023-07-12 20:08:27,153 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497/jenkins-hbase4.apache.org%2C43253%2C1689192505497.meta.1689192507124.meta 2023-07-12 20:08:27,154 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43453,DS-c995922d-4442-40c9-9e1a-e075f395e478,DISK], DatanodeInfoWithStorage[127.0.0.1:41651,DS-8370ea8b-9cf9-4e9e-a7c3-2af949363208,DISK]] 2023-07-12 20:08:27,154 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-12 20:08:27,157 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-12 20:08:27,173 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-12 20:08:27,179 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-12 20:08:27,184 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-12 20:08:27,184 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 20:08:27,184 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-12 20:08:27,184 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-12 20:08:27,188 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 20:08:27,189 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/info 2023-07-12 20:08:27,190 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/info 2023-07-12 20:08:27,190 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 20:08:27,191 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 20:08:27,191 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 20:08:27,193 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/rep_barrier 2023-07-12 20:08:27,193 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/rep_barrier 2023-07-12 20:08:27,194 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 20:08:27,195 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 20:08:27,195 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 20:08:27,197 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/table 2023-07-12 20:08:27,197 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/table 2023-07-12 20:08:27,197 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 20:08:27,198 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 20:08:27,200 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740 2023-07-12 20:08:27,203 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740 2023-07-12 20:08:27,206 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 20:08:27,209 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 20:08:27,210 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68176963, jitterRate=0.015915915369987488}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 20:08:27,211 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 20:08:27,220 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689192507093 2023-07-12 20:08:27,238 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-12 20:08:27,239 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-12 20:08:27,239 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,43253,1689192505497, state=OPEN 2023-07-12 20:08:27,243 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 20:08:27,243 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 20:08:27,247 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-12 20:08:27,247 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,43253,1689192505497 in 335 msec 2023-07-12 20:08:27,253 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-12 20:08:27,253 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 538 msec 2023-07-12 20:08:27,259 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 792 msec 2023-07-12 20:08:27,259 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689192507259, completionTime=-1 2023-07-12 20:08:27,259 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-12 20:08:27,260 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 20:08:27,329 DEBUG [hconnection-0x45a7dedb-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 20:08:27,333 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36948, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 20:08:27,375 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-12 20:08:27,375 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689192567375 2023-07-12 20:08:27,375 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689192627375 2023-07-12 20:08:27,375 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 114 msec 2023-07-12 20:08:27,398 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34965,1689192504325-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:27,398 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34965,1689192504325-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:27,398 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34965,1689192504325-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:27,400 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:34965, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:27,400 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:27,407 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 20:08:27,415 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-12 20:08:27,416 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 20:08:27,426 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-12 20:08:27,428 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 20:08:27,430 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 20:08:27,451 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/.tmp/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:27,454 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/.tmp/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724 empty. 2023-07-12 20:08:27,454 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/.tmp/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:27,455 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-12 20:08:27,487 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-12 20:08:27,489 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => c0bf5c648500555868549c4d8dbe5724, NAME => 'hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/.tmp 2023-07-12 20:08:27,506 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 20:08:27,507 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing c0bf5c648500555868549c4d8dbe5724, disabling compactions & flushes 2023-07-12 20:08:27,507 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:27,507 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:27,507 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. after waiting 0 ms 2023-07-12 20:08:27,507 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:27,507 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:27,507 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for c0bf5c648500555868549c4d8dbe5724: 2023-07-12 20:08:27,511 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 20:08:27,525 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689192507513"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689192507513"}]},"ts":"1689192507513"} 2023-07-12 20:08:27,552 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 20:08:27,554 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 20:08:27,558 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689192507554"}]},"ts":"1689192507554"} 2023-07-12 20:08:27,563 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-12 20:08:27,568 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-12 20:08:27,570 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 20:08:27,570 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 20:08:27,570 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 20:08:27,572 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c0bf5c648500555868549c4d8dbe5724, ASSIGN}] 2023-07-12 20:08:27,575 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c0bf5c648500555868549c4d8dbe5724, ASSIGN 2023-07-12 20:08:27,577 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=c0bf5c648500555868549c4d8dbe5724, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42213,1689192505540; forceNewPlan=false, retain=false 2023-07-12 20:08:27,730 INFO [jenkins-hbase4:34965] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-12 20:08:27,731 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=c0bf5c648500555868549c4d8dbe5724, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42213,1689192505540 2023-07-12 20:08:27,731 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689192507730"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689192507730"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689192507730"}]},"ts":"1689192507730"} 2023-07-12 20:08:27,735 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure c0bf5c648500555868549c4d8dbe5724, server=jenkins-hbase4.apache.org,42213,1689192505540}] 2023-07-12 20:08:27,889 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,42213,1689192505540 2023-07-12 20:08:27,889 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 20:08:27,892 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41088, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 20:08:27,898 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:27,899 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c0bf5c648500555868549c4d8dbe5724, NAME => 'hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724.', STARTKEY => '', ENDKEY => ''} 2023-07-12 20:08:27,900 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:27,900 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 20:08:27,900 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:27,900 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:27,902 INFO [StoreOpener-c0bf5c648500555868549c4d8dbe5724-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:27,904 DEBUG [StoreOpener-c0bf5c648500555868549c4d8dbe5724-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724/info 2023-07-12 20:08:27,905 DEBUG [StoreOpener-c0bf5c648500555868549c4d8dbe5724-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724/info 2023-07-12 20:08:27,905 INFO [StoreOpener-c0bf5c648500555868549c4d8dbe5724-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c0bf5c648500555868549c4d8dbe5724 columnFamilyName info 2023-07-12 20:08:27,906 INFO [StoreOpener-c0bf5c648500555868549c4d8dbe5724-1] regionserver.HStore(310): Store=c0bf5c648500555868549c4d8dbe5724/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 20:08:27,907 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:27,908 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:27,913 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:27,917 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 20:08:27,918 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c0bf5c648500555868549c4d8dbe5724; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73824568, jitterRate=0.10007178783416748}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 20:08:27,918 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c0bf5c648500555868549c4d8dbe5724: 2023-07-12 20:08:27,920 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724., pid=6, masterSystemTime=1689192507889 2023-07-12 20:08:27,925 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:27,925 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:27,927 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=c0bf5c648500555868549c4d8dbe5724, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42213,1689192505540 2023-07-12 20:08:27,927 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689192507926"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689192507926"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689192507926"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689192507926"}]},"ts":"1689192507926"} 2023-07-12 20:08:27,935 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-12 20:08:27,935 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure c0bf5c648500555868549c4d8dbe5724, server=jenkins-hbase4.apache.org,42213,1689192505540 in 196 msec 2023-07-12 20:08:27,938 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-12 20:08:27,939 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=c0bf5c648500555868549c4d8dbe5724, ASSIGN in 363 msec 2023-07-12 20:08:27,940 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 20:08:27,940 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689192507940"}]},"ts":"1689192507940"} 2023-07-12 20:08:27,943 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-12 20:08:27,946 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 20:08:27,949 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 529 msec 2023-07-12 20:08:28,028 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-12 20:08:28,030 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-12 20:08:28,030 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 20:08:28,053 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 20:08:28,055 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41094, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 20:08:28,068 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-12 20:08:28,086 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 20:08:28,093 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 32 msec 2023-07-12 20:08:28,101 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-12 20:08:28,111 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 20:08:28,115 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 14 msec 2023-07-12 20:08:28,127 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 20:08:28,129 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 20:08:28,131 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.530sec 2023-07-12 20:08:28,133 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 20:08:28,135 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 20:08:28,135 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 20:08:28,136 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34965,1689192504325-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 20:08:28,137 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34965,1689192504325-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 20:08:28,145 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 20:08:28,172 DEBUG [Listener at localhost/44421] zookeeper.ReadOnlyZKClient(139): Connect 0x2163cf9f to 127.0.0.1:50668 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 20:08:28,207 DEBUG [Listener at localhost/44421] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6702864a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 20:08:28,221 DEBUG [hconnection-0x6fc7b31a-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 20:08:28,232 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36956, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 20:08:28,241 INFO [Listener at localhost/44421] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,34965,1689192504325 2023-07-12 20:08:28,252 DEBUG [Listener at localhost/44421] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-12 20:08:28,255 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51776, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-12 20:08:28,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34965] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 't', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 20:08:28,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34965] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=t 2023-07-12 20:08:28,270 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 20:08:28,272 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 20:08:28,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34965] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "t" procId is: 9 2023-07-12 20:08:28,274 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/.tmp/data/default/t/4d090c063c934b8812e620114898ad9b 2023-07-12 20:08:28,275 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/.tmp/data/default/t/4d090c063c934b8812e620114898ad9b empty. 2023-07-12 20:08:28,277 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/.tmp/data/default/t/4d090c063c934b8812e620114898ad9b 2023-07-12 20:08:28,277 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived t regions 2023-07-12 20:08:28,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34965] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 20:08:28,296 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/.tmp/data/default/t/.tabledesc/.tableinfo.0000000001 2023-07-12 20:08:28,298 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(7675): creating {ENCODED => 4d090c063c934b8812e620114898ad9b, NAME => 't,,1689192508262.4d090c063c934b8812e620114898ad9b.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='t', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/.tmp 2023-07-12 20:08:28,319 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(866): Instantiated t,,1689192508262.4d090c063c934b8812e620114898ad9b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 20:08:28,319 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1604): Closing 4d090c063c934b8812e620114898ad9b, disabling compactions & flushes 2023-07-12 20:08:28,319 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1626): Closing region t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:28,319 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:28,319 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1714): Acquired close lock on t,,1689192508262.4d090c063c934b8812e620114898ad9b. after waiting 0 ms 2023-07-12 20:08:28,319 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1724): Updates disabled for region t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:28,319 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1838): Closed t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:28,319 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1558): Region close journal for 4d090c063c934b8812e620114898ad9b: 2023-07-12 20:08:28,323 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 20:08:28,325 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"t,,1689192508262.4d090c063c934b8812e620114898ad9b.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689192508325"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689192508325"}]},"ts":"1689192508325"} 2023-07-12 20:08:28,328 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 20:08:28,329 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 20:08:28,329 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689192508329"}]},"ts":"1689192508329"} 2023-07-12 20:08:28,331 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLING in hbase:meta 2023-07-12 20:08:28,336 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-12 20:08:28,337 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 20:08:28,337 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 20:08:28,337 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 20:08:28,337 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=4d090c063c934b8812e620114898ad9b, ASSIGN}] 2023-07-12 20:08:28,339 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=4d090c063c934b8812e620114898ad9b, ASSIGN 2023-07-12 20:08:28,341 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=4d090c063c934b8812e620114898ad9b, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42213,1689192505540; forceNewPlan=false, retain=false 2023-07-12 20:08:28,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34965] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 20:08:28,491 INFO [jenkins-hbase4:34965] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-12 20:08:28,492 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=4d090c063c934b8812e620114898ad9b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42213,1689192505540 2023-07-12 20:08:28,492 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689192508262.4d090c063c934b8812e620114898ad9b.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689192508492"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689192508492"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689192508492"}]},"ts":"1689192508492"} 2023-07-12 20:08:28,495 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 4d090c063c934b8812e620114898ad9b, server=jenkins-hbase4.apache.org,42213,1689192505540}] 2023-07-12 20:08:28,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34965] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 20:08:28,654 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:28,654 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4d090c063c934b8812e620114898ad9b, NAME => 't,,1689192508262.4d090c063c934b8812e620114898ad9b.', STARTKEY => '', ENDKEY => ''} 2023-07-12 20:08:28,655 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 4d090c063c934b8812e620114898ad9b 2023-07-12 20:08:28,655 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689192508262.4d090c063c934b8812e620114898ad9b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 20:08:28,655 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 4d090c063c934b8812e620114898ad9b 2023-07-12 20:08:28,655 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 4d090c063c934b8812e620114898ad9b 2023-07-12 20:08:28,657 INFO [StoreOpener-4d090c063c934b8812e620114898ad9b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 4d090c063c934b8812e620114898ad9b 2023-07-12 20:08:28,659 DEBUG [StoreOpener-4d090c063c934b8812e620114898ad9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b/f 2023-07-12 20:08:28,660 DEBUG [StoreOpener-4d090c063c934b8812e620114898ad9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b/f 2023-07-12 20:08:28,660 INFO [StoreOpener-4d090c063c934b8812e620114898ad9b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4d090c063c934b8812e620114898ad9b columnFamilyName f 2023-07-12 20:08:28,661 INFO [StoreOpener-4d090c063c934b8812e620114898ad9b-1] regionserver.HStore(310): Store=4d090c063c934b8812e620114898ad9b/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 20:08:28,662 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b 2023-07-12 20:08:28,663 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b 2023-07-12 20:08:28,668 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 4d090c063c934b8812e620114898ad9b 2023-07-12 20:08:28,670 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 20:08:28,671 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 4d090c063c934b8812e620114898ad9b; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67930985, jitterRate=0.012250557541847229}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 20:08:28,671 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 4d090c063c934b8812e620114898ad9b: 2023-07-12 20:08:28,672 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689192508262.4d090c063c934b8812e620114898ad9b., pid=11, masterSystemTime=1689192508648 2023-07-12 20:08:28,675 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:28,675 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:28,676 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=4d090c063c934b8812e620114898ad9b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42213,1689192505540 2023-07-12 20:08:28,676 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689192508262.4d090c063c934b8812e620114898ad9b.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689192508675"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689192508675"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689192508675"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689192508675"}]},"ts":"1689192508675"} 2023-07-12 20:08:28,681 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-12 20:08:28,681 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 4d090c063c934b8812e620114898ad9b, server=jenkins-hbase4.apache.org,42213,1689192505540 in 183 msec 2023-07-12 20:08:28,684 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-12 20:08:28,684 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=t, region=4d090c063c934b8812e620114898ad9b, ASSIGN in 344 msec 2023-07-12 20:08:28,685 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 20:08:28,685 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689192508685"}]},"ts":"1689192508685"} 2023-07-12 20:08:28,687 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLED in hbase:meta 2023-07-12 20:08:28,690 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 20:08:28,692 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=t in 424 msec 2023-07-12 20:08:28,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34965] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 20:08:28,892 INFO [Listener at localhost/44421] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:t, procId: 9 completed 2023-07-12 20:08:28,892 DEBUG [Listener at localhost/44421] hbase.HBaseTestingUtility(3430): Waiting until all regions of table t get assigned. Timeout = 60000ms 2023-07-12 20:08:28,893 INFO [Listener at localhost/44421] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 20:08:28,897 INFO [Listener at localhost/44421] hbase.HBaseTestingUtility(3484): All regions for table t assigned to meta. Checking AM states. 2023-07-12 20:08:28,898 INFO [Listener at localhost/44421] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 20:08:28,898 INFO [Listener at localhost/44421] hbase.HBaseTestingUtility(3504): All regions for table t assigned. 2023-07-12 20:08:28,901 DEBUG [Listener at localhost/44421] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 20:08:28,905 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41098, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 20:08:28,951 INFO [Listener at localhost/44421] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686a Thread=294, OpenFileDescriptor=587, MaxFileDescriptor=60000, SystemLoadAverage=233, ProcessCount=170, AvailableMemoryMB=7659 2023-07-12 20:08:28,952 INFO [Listener at localhost/44421] client.TestScannerTimeout(134): START ************ TEST3686A---1 2023-07-12 20:08:28,956 DEBUG [Listener at localhost/44421] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-12 20:08:28,956 DEBUG [Listener at localhost/44421] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:28,957 INFO [Listener at localhost/44421] client.TestScannerTimeout(136): START ************ TEST3686A---1111 2023-07-12 20:08:28,957 INFO [Listener at localhost/44421] client.TestScannerTimeout(140): ************ TEST3686A 2023-07-12 20:08:28,960 INFO [Listener at localhost/44421] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace/table:state/1689192507940/Put/vlen=2/seqid=0} 2023-07-12 20:08:28,960 INFO [Listener at localhost/44421] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=hbase:namespace, state=ENABLED 2023-07-12 20:08:28,960 INFO [Listener at localhost/44421] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724./info:regioninfo/1689192507926/Put/vlen=41/seqid=0, hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724./info:seqnumDuringOpen/1689192507926/Put/vlen=8/seqid=0, hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724./info:server/1689192507926/Put/vlen=31/seqid=0, hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724./info:serverstartcode/1689192507926/Put/vlen=8/seqid=0, hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724./info:sn/1689192507730/Put/vlen=45/seqid=0, hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724./info:state/1689192507926/Put/vlen=4/seqid=0} 2023-07-12 20:08:28,960 INFO [Listener at localhost/44421] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => c0bf5c648500555868549c4d8dbe5724, NAME => 'hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724.', STARTKEY => '', ENDKEY => ''} 2023-07-12 20:08:28,960 INFO [Listener at localhost/44421] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t/table:state/1689192508685/Put/vlen=2/seqid=0} 2023-07-12 20:08:28,961 INFO [Listener at localhost/44421] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=t, state=ENABLED 2023-07-12 20:08:28,961 INFO [Listener at localhost/44421] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t,,1689192508262.4d090c063c934b8812e620114898ad9b./info:regioninfo/1689192508675/Put/vlen=35/seqid=0, t,,1689192508262.4d090c063c934b8812e620114898ad9b./info:seqnumDuringOpen/1689192508675/Put/vlen=8/seqid=0, t,,1689192508262.4d090c063c934b8812e620114898ad9b./info:server/1689192508675/Put/vlen=31/seqid=0, t,,1689192508262.4d090c063c934b8812e620114898ad9b./info:serverstartcode/1689192508675/Put/vlen=8/seqid=0, t,,1689192508262.4d090c063c934b8812e620114898ad9b./info:sn/1689192508492/Put/vlen=45/seqid=0, t,,1689192508262.4d090c063c934b8812e620114898ad9b./info:state/1689192508675/Put/vlen=4/seqid=0} 2023-07-12 20:08:28,961 INFO [Listener at localhost/44421] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => 4d090c063c934b8812e620114898ad9b, NAME => 't,,1689192508262.4d090c063c934b8812e620114898ad9b.', STARTKEY => '', ENDKEY => ''} 2023-07-12 20:08:28,962 DEBUG [Listener at localhost/44421] zookeeper.ReadOnlyZKClient(139): Connect 0x3b8de6b0 to 127.0.0.1:50668 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 20:08:28,966 DEBUG [Listener at localhost/44421] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46b319a0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 20:08:28,966 INFO [Listener at localhost/44421] client.TestScannerTimeout(150): START ************ TEST3686A---22 2023-07-12 20:08:28,967 INFO [Listener at localhost/44421] client.TestScannerTimeout(153): START ************ TEST3686A---33 2023-07-12 20:08:28,969 DEBUG [hconnection-0x2e74ef4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 20:08:28,972 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36958, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 20:08:28,976 DEBUG [hconnection-0x2e74ef4-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 20:08:28,979 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41106, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 20:08:28,980 INFO [Listener at localhost/44421] client.TestScannerTimeout(157): START ************ TEST3686A---44 2023-07-12 20:08:28,983 ERROR [Listener at localhost/44421] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,42213,1689192505540: die! ***** 2023-07-12 20:08:28,984 ERROR [Listener at localhost/44421] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 20:08:28,986 DEBUG [Listener at localhost/44421] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-12 20:08:28,989 DEBUG [Listener at localhost/44421] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-12 20:08:28,989 DEBUG [Listener at localhost/44421] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-12 20:08:28,990 DEBUG [Listener at localhost/44421] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-12 20:08:28,990 INFO [Listener at localhost/44421] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 965738496, "init": 513802240, "max": 2051014656, "used": 172479912 }, "NonHeapMemoryUsage": { "committed": 113074176, "init": 2555904, "max": -1, "used": 111021584 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ] } 2023-07-12 20:08:28,994 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34965] master.MasterRpcServices(612): jenkins-hbase4.apache.org,42213,1689192505540 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,42213,1689192505540: die! ***** 2023-07-12 20:08:28,996 INFO [Listener at localhost/44421] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,42213,1689192505540' ***** 2023-07-12 20:08:28,997 INFO [Listener at localhost/44421] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-12 20:08:28,997 INFO [RS:1;jenkins-hbase4:42213] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 20:08:28,997 INFO [RS:1;jenkins-hbase4:42213] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-12 20:08:28,997 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 20:08:28,997 INFO [RS:1;jenkins-hbase4:42213] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-12 20:08:28,998 INFO [RS:1;jenkins-hbase4:42213] regionserver.HRegionServer(3305): Received CLOSE for c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:28,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42213] regionserver.RSRpcServices(3499): Server shutting down and client tried to access missing scanner 2450164437324136450 2023-07-12 20:08:28,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42213] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,42213,1689192505540 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 20:08:28,998 INFO [RS:1;jenkins-hbase4:42213] regionserver.HRegionServer(3305): Received CLOSE for 4d090c063c934b8812e620114898ad9b 2023-07-12 20:08:28,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42213] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 172.31.14.131:41106 deadline: 1689192568998, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,42213,1689192505540 aborting 2023-07-12 20:08:28,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c0bf5c648500555868549c4d8dbe5724, disabling compactions & flushes 2023-07-12 20:08:28,999 INFO [RS:1;jenkins-hbase4:42213] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,42213,1689192505540 2023-07-12 20:08:28,999 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:28,999 DEBUG [RS:1;jenkins-hbase4:42213] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x168a8e12 to 127.0.0.1:50668 2023-07-12 20:08:28,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:28,999 DEBUG [RS:1;jenkins-hbase4:42213] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 20:08:28,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. after waiting 0 ms 2023-07-12 20:08:29,000 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:29,000 INFO [RS:1;jenkins-hbase4:42213] regionserver.HRegionServer(1474): Waiting on 2 regions to close 2023-07-12 20:08:29,000 DEBUG [RS:1;jenkins-hbase4:42213] regionserver.HRegionServer(1478): Online Regions={c0bf5c648500555868549c4d8dbe5724=hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724., 4d090c063c934b8812e620114898ad9b=t,,1689192508262.4d090c063c934b8812e620114898ad9b.} 2023-07-12 20:08:29,000 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:29,001 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:29,001 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c0bf5c648500555868549c4d8dbe5724: 2023-07-12 20:08:29,001 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:29,001 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 4d090c063c934b8812e620114898ad9b, disabling compactions & flushes 2023-07-12 20:08:29,001 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:29,001 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:29,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689192508262.4d090c063c934b8812e620114898ad9b. after waiting 0 ms 2023-07-12 20:08:29,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:29,002 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 270 in region t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:29,002 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:29,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 4d090c063c934b8812e620114898ad9b: 2023-07-12 20:08:29,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:29,003 DEBUG [RS:1;jenkins-hbase4:42213] regionserver.HRegionServer(1504): Waiting on 2023-07-12 20:08:29,016 DEBUG [Listener at localhost/44421] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:232) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,42213,1689192505540 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) ... 11 more Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException): org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,42213,1689192505540 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-12 20:08:29,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42213] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,42213,1689192505540 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 20:08:29,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42213] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 125 connection: 172.31.14.131:41106 deadline: 1689192569018, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,42213,1689192505540 aborting 2023-07-12 20:08:29,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42213] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,42213,1689192505540 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 20:08:29,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42213] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Scan size: 125 connection: 172.31.14.131:41106 deadline: 1689192569126, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,42213,1689192505540 aborting 2023-07-12 20:08:29,203 INFO [RS:1;jenkins-hbase4:42213] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,42213,1689192505540; all regions closed. 2023-07-12 20:08:29,212 DEBUG [RS:1;jenkins-hbase4:42213] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 20:08:29,213 INFO [RS:1;jenkins-hbase4:42213] regionserver.LeaseManager(133): Closed leases 2023-07-12 20:08:29,213 INFO [RS:1;jenkins-hbase4:42213] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-12 20:08:29,213 INFO [RS:1;jenkins-hbase4:42213] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 20:08:29,213 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 20:08:29,213 INFO [RS:1;jenkins-hbase4:42213] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 20:08:29,214 INFO [RS:1;jenkins-hbase4:42213] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 20:08:29,214 INFO [RS:1;jenkins-hbase4:42213] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42213 2023-07-12 20:08:29,225 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 20:08:29,225 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): regionserver:43253-0x1015b269bb50001, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42213,1689192505540 2023-07-12 20:08:29,225 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): regionserver:42213-0x1015b269bb50002, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42213,1689192505540 2023-07-12 20:08:29,225 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): regionserver:43253-0x1015b269bb50001, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 20:08:29,226 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): regionserver:42213-0x1015b269bb50002, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 20:08:29,226 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,42213,1689192505540] 2023-07-12 20:08:29,227 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,42213,1689192505540; numProcessing=1 2023-07-12 20:08:29,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43253-0x1015b269bb50001, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43253,1689192505497 2023-07-12 20:08:29,228 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,42213,1689192505540 znode expired, triggering replicatorRemoved event 2023-07-12 20:08:29,229 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,42213,1689192505540 already deleted, retry=false 2023-07-12 20:08:29,229 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,42213,1689192505540 on jenkins-hbase4.apache.org,34965,1689192504325 2023-07-12 20:08:29,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43253-0x1015b269bb50001, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43253,1689192505497 2023-07-12 20:08:29,237 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,42213,1689192505540, splitWal=true, meta=false 2023-07-12 20:08:29,237 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=12 for jenkins-hbase4.apache.org,42213,1689192505540 (carryingMeta=false) jenkins-hbase4.apache.org,42213,1689192505540/CRASHED/regionCount=2/lock=java.util.concurrent.locks.ReentrantReadWriteLock@52ee953b[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-12 20:08:29,240 INFO [PEWorker-4] procedure.ServerCrashProcedure(161): Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,42213,1689192505540, splitWal=true, meta=false 2023-07-12 20:08:29,241 INFO [PEWorker-4] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,42213,1689192505540 had 2 regions 2023-07-12 20:08:29,243 INFO [PEWorker-4] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,42213,1689192505540, splitWal=true, meta=false, isMeta: false 2023-07-12 20:08:29,245 DEBUG [PEWorker-4] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,42213,1689192505540-splitting 2023-07-12 20:08:29,246 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,42213,1689192505540 WAL count=1, meta=false 2023-07-12 20:08:29,249 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894}] 2023-07-12 20:08:29,256 DEBUG [PEWorker-5] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,43253,1689192505497 2023-07-12 20:08:29,258 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894, worker=jenkins-hbase4.apache.org,43253,1689192505497}] 2023-07-12 20:08:29,328 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): regionserver:42213-0x1015b269bb50002, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 20:08:29,329 INFO [RS:1;jenkins-hbase4:42213] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,42213,1689192505540; zookeeper connection closed. 2023-07-12 20:08:29,329 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): regionserver:42213-0x1015b269bb50002, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 20:08:29,329 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@517ee26b] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@517ee26b 2023-07-12 20:08:29,335 WARN [RPCClient-NioEventLoopGroup-5-5] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:42213 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42213 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 20:08:29,336 DEBUG [RPCClient-NioEventLoopGroup-5-5] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:42213 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42213 2023-07-12 20:08:29,421 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=43253] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=14 2023-07-12 20:08:29,443 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,42213,1689192505540-splitting/jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894, size=1.6 K (1665bytes) 2023-07-12 20:08:29,444 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,42213,1689192505540-splitting/jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894 2023-07-12 20:08:29,449 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,42213,1689192505540-splitting/jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894 after 5ms 2023-07-12 20:08:29,454 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,42213,1689192505540-splitting/jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894 took 12ms 2023-07-12 20:08:29,468 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for c0bf5c648500555868549c4d8dbe5724: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-12 20:08:29,471 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 4d090c063c934b8812e620114898ad9b: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 3 } 2023-07-12 20:08:29,472 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,42213,1689192505540-splitting/jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894 so closing down 2023-07-12 20:08:29,472 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-12 20:08:29,474 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-12 20:08:29,477 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-1] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894.temp 2023-07-12 20:08:29,477 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894.temp 2023-07-12 20:08:29,478 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-12 20:08:29,497 INFO [split-log-closeStream-pool-1] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-12 20:08:29,498 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-12 20:08:29,499 INFO [split-log-closeStream-pool-1] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894.temp to hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b/recovered.edits/0000000000000000013 2023-07-12 20:08:29,500 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894.temp to hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724/recovered.edits/0000000000000000005 2023-07-12 20:08:29,501 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 14 edits across 2 Regions in 45 ms; skipped=2; WAL=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,42213,1689192505540-splitting/jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894, size=1.6 K, length=1665, corrupted=false, cancelled=false 2023-07-12 20:08:29,501 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,42213,1689192505540-splitting/jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894, journal: Splitting hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,42213,1689192505540-splitting/jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894, size=1.6 K (1665bytes) at 1689192509444Finishing writing output for hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,42213,1689192505540-splitting/jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894 so closing down at 1689192509472 (+28 ms)Creating recovered edits writer path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894.temp at 1689192509477 (+5 ms)Creating recovered edits writer path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894.temp at 16891925094773 split writer threads finished at 1689192509478 (+1 ms)Closed recovered edits writer path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689192509497 (+19 ms)Closed recovered edits writer path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689192509498 (+1 ms)Rename recovered edits hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894.temp to hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b/recovered.edits/0000000000000000013 at 1689192509500 (+2 ms)Rename recovered edits hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894.temp to hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724/recovered.edits/0000000000000000005 at 1689192509500Processed 14 edits across 2 Regions in 45 ms; skipped=2; WAL=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,42213,1689192505540-splitting/jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894, size=1.6 K, length=1665, corrupted=false, cancelled=false at 1689192509501 (+1 ms) 2023-07-12 20:08:29,501 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,42213,1689192505540-splitting/jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894 2023-07-12 20:08:29,503 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=14 2023-07-12 20:08:29,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34965] master.HMaster(3641): Remote procedure done, pid=14 2023-07-12 20:08:29,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34965] wal.WALSplitUtil(141): Moved hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,42213,1689192505540-splitting/jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894 to hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/oldWALs 2023-07-12 20:08:29,516 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-12 20:08:29,516 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894, worker=jenkins-hbase4.apache.org,43253,1689192505497 in 255 msec 2023-07-12 20:08:29,517 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,43253,1689192505497 2023-07-12 20:08:29,521 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-12 20:08:29,521 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C42213%2C1689192505540.1689192506894, worker=jenkins-hbase4.apache.org,43253,1689192505497 in 270 msec 2023-07-12 20:08:29,523 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,42213,1689192505540-splitting dir is empty, no logs to split. 2023-07-12 20:08:29,523 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,42213,1689192505540 WAL count=0, meta=false 2023-07-12 20:08:29,523 DEBUG [PEWorker-4] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,42213,1689192505540 WAL splitting is done? wals=0, meta=false 2023-07-12 20:08:29,526 INFO [PEWorker-4] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase4.apache.org,42213,1689192505540 failed, ignore...File hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,42213,1689192505540-splitting does not exist. 2023-07-12 20:08:29,528 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=4d090c063c934b8812e620114898ad9b, ASSIGN}, {pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c0bf5c648500555868549c4d8dbe5724, ASSIGN}] 2023-07-12 20:08:29,529 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c0bf5c648500555868549c4d8dbe5724, ASSIGN 2023-07-12 20:08:29,529 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=4d090c063c934b8812e620114898ad9b, ASSIGN 2023-07-12 20:08:29,530 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=c0bf5c648500555868549c4d8dbe5724, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-12 20:08:29,530 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=4d090c063c934b8812e620114898ad9b, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-12 20:08:29,640 DEBUG [hconnection-0x2e74ef4-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:42213 this server is in the failed servers list 2023-07-12 20:08:29,681 DEBUG [jenkins-hbase4:34965] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-12 20:08:29,681 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=c0bf5c648500555868549c4d8dbe5724, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43253,1689192505497 2023-07-12 20:08:29,681 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=4d090c063c934b8812e620114898ad9b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43253,1689192505497 2023-07-12 20:08:29,681 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689192509681"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689192509681"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689192509681"}]},"ts":"1689192509681"} 2023-07-12 20:08:29,681 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689192508262.4d090c063c934b8812e620114898ad9b.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689192509681"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689192509681"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689192509681"}]},"ts":"1689192509681"} 2023-07-12 20:08:29,684 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE; OpenRegionProcedure c0bf5c648500555868549c4d8dbe5724, server=jenkins-hbase4.apache.org,43253,1689192505497}] 2023-07-12 20:08:29,685 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=15, state=RUNNABLE; OpenRegionProcedure 4d090c063c934b8812e620114898ad9b, server=jenkins-hbase4.apache.org,43253,1689192505497}] 2023-07-12 20:08:29,735 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 20:08:29,842 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:29,842 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4d090c063c934b8812e620114898ad9b, NAME => 't,,1689192508262.4d090c063c934b8812e620114898ad9b.', STARTKEY => '', ENDKEY => ''} 2023-07-12 20:08:29,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 4d090c063c934b8812e620114898ad9b 2023-07-12 20:08:29,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689192508262.4d090c063c934b8812e620114898ad9b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 20:08:29,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 4d090c063c934b8812e620114898ad9b 2023-07-12 20:08:29,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 4d090c063c934b8812e620114898ad9b 2023-07-12 20:08:29,845 INFO [StoreOpener-4d090c063c934b8812e620114898ad9b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 4d090c063c934b8812e620114898ad9b 2023-07-12 20:08:29,846 DEBUG [StoreOpener-4d090c063c934b8812e620114898ad9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b/f 2023-07-12 20:08:29,847 DEBUG [StoreOpener-4d090c063c934b8812e620114898ad9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b/f 2023-07-12 20:08:29,847 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:29,847 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c0bf5c648500555868549c4d8dbe5724, NAME => 'hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724.', STARTKEY => '', ENDKEY => ''} 2023-07-12 20:08:29,847 INFO [StoreOpener-4d090c063c934b8812e620114898ad9b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4d090c063c934b8812e620114898ad9b columnFamilyName f 2023-07-12 20:08:29,848 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:29,848 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 20:08:29,848 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:29,848 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:29,848 INFO [StoreOpener-4d090c063c934b8812e620114898ad9b-1] regionserver.HStore(310): Store=4d090c063c934b8812e620114898ad9b/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 20:08:29,850 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b 2023-07-12 20:08:29,851 INFO [StoreOpener-c0bf5c648500555868549c4d8dbe5724-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:29,852 DEBUG [StoreOpener-c0bf5c648500555868549c4d8dbe5724-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724/info 2023-07-12 20:08:29,852 DEBUG [StoreOpener-c0bf5c648500555868549c4d8dbe5724-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724/info 2023-07-12 20:08:29,853 INFO [StoreOpener-c0bf5c648500555868549c4d8dbe5724-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c0bf5c648500555868549c4d8dbe5724 columnFamilyName info 2023-07-12 20:08:29,853 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b 2023-07-12 20:08:29,853 INFO [StoreOpener-c0bf5c648500555868549c4d8dbe5724-1] regionserver.HStore(310): Store=c0bf5c648500555868549c4d8dbe5724/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 20:08:29,855 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b/recovered.edits/0000000000000000013 2023-07-12 20:08:29,855 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:29,857 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:29,859 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724/recovered.edits/0000000000000000005 2023-07-12 20:08:29,862 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 10, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b/recovered.edits/0000000000000000013 2023-07-12 20:08:29,862 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 4d090c063c934b8812e620114898ad9b 1/1 column families, dataSize=270 B heapSize=1.27 KB 2023-07-12 20:08:29,863 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724/recovered.edits/0000000000000000005 2023-07-12 20:08:29,863 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing c0bf5c648500555868549c4d8dbe5724 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-12 20:08:29,922 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724/.tmp/info/918473fcd52d467483dc49baa2aece6d 2023-07-12 20:08:29,922 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=270 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b/.tmp/f/fadadbb9f85b49099db7f2fa514cdd64 2023-07-12 20:08:29,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b/.tmp/f/fadadbb9f85b49099db7f2fa514cdd64 as hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b/f/fadadbb9f85b49099db7f2fa514cdd64 2023-07-12 20:08:29,966 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724/.tmp/info/918473fcd52d467483dc49baa2aece6d as hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724/info/918473fcd52d467483dc49baa2aece6d 2023-07-12 20:08:29,975 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b/f/fadadbb9f85b49099db7f2fa514cdd64, entries=10, sequenceid=13, filesize=4.8 K 2023-07-12 20:08:29,975 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724/info/918473fcd52d467483dc49baa2aece6d, entries=2, sequenceid=5, filesize=4.8 K 2023-07-12 20:08:29,976 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for c0bf5c648500555868549c4d8dbe5724 in 113ms, sequenceid=5, compaction requested=false; wal=null 2023-07-12 20:08:29,976 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~270 B/270, heapSize ~1.25 KB/1280, currentSize=0 B/0 for 4d090c063c934b8812e620114898ad9b in 114ms, sequenceid=13, compaction requested=false; wal=null 2023-07-12 20:08:29,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 't' 2023-07-12 20:08:29,977 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-12 20:08:29,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b/recovered.edits/0000000000000000013 2023-07-12 20:08:29,981 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724/recovered.edits/0000000000000000005 2023-07-12 20:08:29,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 4d090c063c934b8812e620114898ad9b 2023-07-12 20:08:29,987 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:29,990 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-12 20:08:29,991 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2023-07-12 20:08:29,992 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 4d090c063c934b8812e620114898ad9b; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65906951, jitterRate=-0.017909899353981018}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 20:08:29,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 4d090c063c934b8812e620114898ad9b: 2023-07-12 20:08:29,992 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c0bf5c648500555868549c4d8dbe5724; next sequenceid=6; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68390722, jitterRate=0.01910117268562317}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 20:08:29,992 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c0bf5c648500555868549c4d8dbe5724: 2023-07-12 20:08:29,993 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689192508262.4d090c063c934b8812e620114898ad9b., pid=18, masterSystemTime=1689192509836 2023-07-12 20:08:29,994 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724., pid=17, masterSystemTime=1689192509836 2023-07-12 20:08:29,996 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:29,996 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:29,997 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=c0bf5c648500555868549c4d8dbe5724, regionState=OPEN, openSeqNum=6, regionLocation=jenkins-hbase4.apache.org,43253,1689192505497 2023-07-12 20:08:29,997 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689192509996"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689192509996"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689192509996"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689192509996"}]},"ts":"1689192509996"} 2023-07-12 20:08:29,997 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:29,997 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:29,998 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=4d090c063c934b8812e620114898ad9b, regionState=OPEN, openSeqNum=14, regionLocation=jenkins-hbase4.apache.org,43253,1689192505497 2023-07-12 20:08:29,999 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689192508262.4d090c063c934b8812e620114898ad9b.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689192509998"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689192509998"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689192509998"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689192509998"}]},"ts":"1689192509998"} 2023-07-12 20:08:30,003 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-07-12 20:08:30,004 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; OpenRegionProcedure c0bf5c648500555868549c4d8dbe5724, server=jenkins-hbase4.apache.org,43253,1689192505497 in 316 msec 2023-07-12 20:08:30,005 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=15 2023-07-12 20:08:30,005 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=15, state=SUCCESS; OpenRegionProcedure 4d090c063c934b8812e620114898ad9b, server=jenkins-hbase4.apache.org,43253,1689192505497 in 317 msec 2023-07-12 20:08:30,006 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=c0bf5c648500555868549c4d8dbe5724, ASSIGN in 476 msec 2023-07-12 20:08:30,009 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-07-12 20:08:30,009 INFO [PEWorker-5] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,42213,1689192505540 after splitting done 2023-07-12 20:08:30,009 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=t, region=4d090c063c934b8812e620114898ad9b, ASSIGN in 477 msec 2023-07-12 20:08:30,009 DEBUG [PEWorker-5] master.DeadServer(114): Removed jenkins-hbase4.apache.org,42213,1689192505540 from processing; numProcessing=0 2023-07-12 20:08:30,012 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,42213,1689192505540, splitWal=true, meta=false in 778 msec 2023-07-12 20:08:30,161 DEBUG [Listener at localhost/44421] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b8de6b0 to 127.0.0.1:50668 2023-07-12 20:08:30,162 DEBUG [Listener at localhost/44421] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 20:08:30,162 INFO [Listener at localhost/44421] client.TestScannerTimeout(168): ************ END TEST3686A 2023-07-12 20:08:30,175 INFO [Listener at localhost/44421] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686a Thread=273 (was 294), OpenFileDescriptor=577 (was 587), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=233 (was 233), ProcessCount=170 (was 170), AvailableMemoryMB=7652 (was 7659) 2023-07-12 20:08:30,185 INFO [Listener at localhost/44421] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686b Thread=273, OpenFileDescriptor=577, MaxFileDescriptor=60000, SystemLoadAverage=233, ProcessCount=170, AvailableMemoryMB=7651 2023-07-12 20:08:30,186 INFO [Listener at localhost/44421] hbase.LocalHBaseCluster(262): Not alive RS:1;jenkins-hbase4:42213 2023-07-12 20:08:30,197 INFO [Listener at localhost/44421] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-12 20:08:30,198 INFO [Listener at localhost/44421] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 20:08:30,198 INFO [Listener at localhost/44421] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 20:08:30,198 INFO [Listener at localhost/44421] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 20:08:30,198 INFO [Listener at localhost/44421] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 20:08:30,198 INFO [Listener at localhost/44421] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 20:08:30,198 INFO [Listener at localhost/44421] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 20:08:30,200 INFO [Listener at localhost/44421] ipc.NettyRpcServer(120): Bind to /172.31.14.131:37151 2023-07-12 20:08:30,201 INFO [Listener at localhost/44421] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 20:08:30,201 DEBUG [Listener at localhost/44421] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 20:08:30,202 INFO [Listener at localhost/44421] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 20:08:30,203 INFO [Listener at localhost/44421] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 20:08:30,204 INFO [Listener at localhost/44421] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:37151 connecting to ZooKeeper ensemble=127.0.0.1:50668 2023-07-12 20:08:30,207 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): regionserver:371510x0, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 20:08:30,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:37151-0x1015b269bb50008 connected 2023-07-12 20:08:30,209 DEBUG [Listener at localhost/44421] zookeeper.ZKUtil(162): regionserver:37151-0x1015b269bb50008, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 20:08:30,210 DEBUG [Listener at localhost/44421] zookeeper.ZKUtil(162): regionserver:37151-0x1015b269bb50008, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-12 20:08:30,211 DEBUG [Listener at localhost/44421] zookeeper.ZKUtil(164): regionserver:37151-0x1015b269bb50008, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 20:08:30,211 DEBUG [Listener at localhost/44421] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37151 2023-07-12 20:08:30,211 DEBUG [Listener at localhost/44421] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37151 2023-07-12 20:08:30,211 DEBUG [Listener at localhost/44421] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37151 2023-07-12 20:08:30,212 DEBUG [Listener at localhost/44421] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37151 2023-07-12 20:08:30,212 DEBUG [Listener at localhost/44421] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37151 2023-07-12 20:08:30,213 INFO [RS:2;jenkins-hbase4:37151] regionserver.HRegionServer(951): ClusterId : 0fd096bf-885b-42ee-9942-e5df6a803c4e 2023-07-12 20:08:30,214 DEBUG [RS:2;jenkins-hbase4:37151] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 20:08:30,216 DEBUG [RS:2;jenkins-hbase4:37151] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 20:08:30,216 DEBUG [RS:2;jenkins-hbase4:37151] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 20:08:30,218 DEBUG [RS:2;jenkins-hbase4:37151] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 20:08:30,219 DEBUG [RS:2;jenkins-hbase4:37151] zookeeper.ReadOnlyZKClient(139): Connect 0x314c4d29 to 127.0.0.1:50668 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 20:08:30,223 DEBUG [RS:2;jenkins-hbase4:37151] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f8bb570, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 20:08:30,223 DEBUG [RS:2;jenkins-hbase4:37151] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3bb42659, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-12 20:08:30,231 DEBUG [RS:2;jenkins-hbase4:37151] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:37151 2023-07-12 20:08:30,232 INFO [RS:2;jenkins-hbase4:37151] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 20:08:30,232 INFO [RS:2;jenkins-hbase4:37151] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 20:08:30,232 DEBUG [RS:2;jenkins-hbase4:37151] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 20:08:30,232 INFO [RS:2;jenkins-hbase4:37151] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,34965,1689192504325 with isa=jenkins-hbase4.apache.org/172.31.14.131:37151, startcode=1689192510197 2023-07-12 20:08:30,232 DEBUG [RS:2;jenkins-hbase4:37151] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 20:08:30,235 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48519, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 20:08:30,235 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=34965] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,37151,1689192510197 2023-07-12 20:08:30,236 DEBUG [RS:2;jenkins-hbase4:37151] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4 2023-07-12 20:08:30,236 DEBUG [RS:2;jenkins-hbase4:37151] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:45739 2023-07-12 20:08:30,236 DEBUG [RS:2;jenkins-hbase4:37151] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 20:08:30,238 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): regionserver:43253-0x1015b269bb50001, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 20:08:30,238 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 20:08:30,238 DEBUG [RS:2;jenkins-hbase4:37151] zookeeper.ZKUtil(162): regionserver:37151-0x1015b269bb50008, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37151,1689192510197 2023-07-12 20:08:30,238 WARN [RS:2;jenkins-hbase4:37151] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 20:08:30,238 INFO [RS:2;jenkins-hbase4:37151] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 20:08:30,239 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,37151,1689192510197] 2023-07-12 20:08:30,239 DEBUG [RS:2;jenkins-hbase4:37151] regionserver.HRegionServer(1948): logDir=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,37151,1689192510197 2023-07-12 20:08:30,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43253-0x1015b269bb50001, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37151,1689192510197 2023-07-12 20:08:30,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43253-0x1015b269bb50001, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43253,1689192505497 2023-07-12 20:08:30,245 DEBUG [RS:2;jenkins-hbase4:37151] zookeeper.ZKUtil(162): regionserver:37151-0x1015b269bb50008, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37151,1689192510197 2023-07-12 20:08:30,245 DEBUG [RS:2;jenkins-hbase4:37151] zookeeper.ZKUtil(162): regionserver:37151-0x1015b269bb50008, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43253,1689192505497 2023-07-12 20:08:30,246 DEBUG [RS:2;jenkins-hbase4:37151] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 20:08:30,247 INFO [RS:2;jenkins-hbase4:37151] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 20:08:30,248 INFO [RS:2;jenkins-hbase4:37151] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 20:08:30,249 INFO [RS:2;jenkins-hbase4:37151] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 20:08:30,249 INFO [RS:2;jenkins-hbase4:37151] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:30,249 INFO [RS:2;jenkins-hbase4:37151] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 20:08:30,250 INFO [RS:2;jenkins-hbase4:37151] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:30,251 DEBUG [RS:2;jenkins-hbase4:37151] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:30,251 DEBUG [RS:2;jenkins-hbase4:37151] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:30,251 DEBUG [RS:2;jenkins-hbase4:37151] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:30,251 DEBUG [RS:2;jenkins-hbase4:37151] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:30,251 DEBUG [RS:2;jenkins-hbase4:37151] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:30,251 DEBUG [RS:2;jenkins-hbase4:37151] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-12 20:08:30,251 DEBUG [RS:2;jenkins-hbase4:37151] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:30,251 DEBUG [RS:2;jenkins-hbase4:37151] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:30,251 DEBUG [RS:2;jenkins-hbase4:37151] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:30,251 DEBUG [RS:2;jenkins-hbase4:37151] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 20:08:30,253 INFO [RS:2;jenkins-hbase4:37151] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:30,253 INFO [RS:2;jenkins-hbase4:37151] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:30,253 INFO [RS:2;jenkins-hbase4:37151] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:30,264 INFO [RS:2;jenkins-hbase4:37151] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 20:08:30,264 INFO [RS:2;jenkins-hbase4:37151] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37151,1689192510197-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 20:08:30,277 INFO [RS:2;jenkins-hbase4:37151] regionserver.Replication(203): jenkins-hbase4.apache.org,37151,1689192510197 started 2023-07-12 20:08:30,277 INFO [RS:2;jenkins-hbase4:37151] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,37151,1689192510197, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:37151, sessionid=0x1015b269bb50008 2023-07-12 20:08:30,277 INFO [Listener at localhost/44421] hbase.HBaseTestingUtility(3254): Started new server=Thread[RS:2;jenkins-hbase4:37151,5,FailOnTimeoutGroup] 2023-07-12 20:08:30,277 DEBUG [RS:2;jenkins-hbase4:37151] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 20:08:30,277 DEBUG [RS:2;jenkins-hbase4:37151] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,37151,1689192510197 2023-07-12 20:08:30,277 DEBUG [RS:2;jenkins-hbase4:37151] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,37151,1689192510197' 2023-07-12 20:08:30,277 DEBUG [RS:2;jenkins-hbase4:37151] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 20:08:30,277 INFO [Listener at localhost/44421] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,42213,1689192505540 2023-07-12 20:08:30,278 INFO [Listener at localhost/44421] client.TestScannerTimeout(177): START ************ test3686b 2023-07-12 20:08:30,278 DEBUG [RS:2;jenkins-hbase4:37151] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 20:08:30,279 DEBUG [RS:2;jenkins-hbase4:37151] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 20:08:30,279 DEBUG [RS:2;jenkins-hbase4:37151] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 20:08:30,279 DEBUG [RS:2;jenkins-hbase4:37151] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,37151,1689192510197 2023-07-12 20:08:30,279 DEBUG [RS:2;jenkins-hbase4:37151] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,37151,1689192510197' 2023-07-12 20:08:30,279 DEBUG [RS:2;jenkins-hbase4:37151] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 20:08:30,279 DEBUG [RS:2;jenkins-hbase4:37151] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 20:08:30,280 DEBUG [RS:2;jenkins-hbase4:37151] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 20:08:30,280 INFO [RS:2;jenkins-hbase4:37151] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 20:08:30,280 INFO [RS:2;jenkins-hbase4:37151] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 20:08:30,281 DEBUG [Listener at localhost/44421] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-12 20:08:30,281 DEBUG [Listener at localhost/44421] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:30,281 DEBUG [Listener at localhost/44421] zookeeper.ReadOnlyZKClient(139): Connect 0x5256e71c to 127.0.0.1:50668 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 20:08:30,285 DEBUG [Listener at localhost/44421] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d47dd46, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 20:08:30,289 DEBUG [hconnection-0x626aa4ba-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 20:08:30,291 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36960, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 20:08:30,382 INFO [RS:2;jenkins-hbase4:37151] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C37151%2C1689192510197, suffix=, logDir=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,37151,1689192510197, archiveDir=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/oldWALs, maxLogs=32 2023-07-12 20:08:30,400 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43453,DS-c995922d-4442-40c9-9e1a-e075f395e478,DISK] 2023-07-12 20:08:30,400 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41651,DS-8370ea8b-9cf9-4e9e-a7c3-2af949363208,DISK] 2023-07-12 20:08:30,405 INFO [RS:2;jenkins-hbase4:37151] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,37151,1689192510197/jenkins-hbase4.apache.org%2C37151%2C1689192510197.1689192510383 2023-07-12 20:08:30,405 DEBUG [RS:2;jenkins-hbase4:37151] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43453,DS-c995922d-4442-40c9-9e1a-e075f395e478,DISK], DatanodeInfoWithStorage[127.0.0.1:41651,DS-8370ea8b-9cf9-4e9e-a7c3-2af949363208,DISK]] 2023-07-12 20:08:32,695 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 20:08:32,758 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-12 20:08:34,489 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 20:08:34,489 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-12 20:08:45,295 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.RSRpcServices$ScannerListener(511): Scanner lease 48687581763731474 expired clientIPAndPort=172.31.14.131:36960, userName=jenkins, regionInfo=t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:47,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43253] regionserver.RSRpcServices(3094): Client tried to access missing scanner 48687581763731474 2023-07-12 20:08:47,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43253] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 41 connection: 172.31.14.131:36960 deadline: 1689192587297, exception=org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '48687581763731474'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. 2023-07-12 20:08:47,299 DEBUG [Listener at localhost/44421] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '48687581763731474'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '48687581763731474'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '48687581763731474'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-12 20:08:47,302 DEBUG [Listener at localhost/44421] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5256e71c to 127.0.0.1:50668 2023-07-12 20:08:47,302 DEBUG [Listener at localhost/44421] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 20:08:47,302 INFO [Listener at localhost/44421] client.TestScannerTimeout(201): END ************ END test3686b 2023-07-12 20:08:47,318 INFO [Listener at localhost/44421] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686b Thread=296 (was 273) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/cluster_b448c56c-3a9d-1fce-b944-8d379dcf0ce4/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/cluster_b448c56c-3a9d-1fce-b944-8d379dcf0ce4/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS:2;jenkins-hbase4:37151-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:50668@0x314c4d29-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:45739 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:50668@0x5256e71c-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:50668@0x5256e71c java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1481) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/362786809.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4-prefix:jenkins-hbase4.apache.org,37151,1689192510197 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-1149810363-172.31.14.131-1689192501335:blk_1073741843_1019, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase4:37151 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/cluster_b448c56c-3a9d-1fce-b944-8d379dcf0ce4/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-915468328_17 at /127.0.0.1:57540 [Receiving block BP-1149810363-172.31.14.131-1689192501335:blk_1073741843_1019] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/cluster_b448c56c-3a9d-1fce-b944-8d379dcf0ce4/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-1149810363-172.31.14.131-1689192501335:blk_1073741843_1019, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:50668@0x314c4d29-SendThread(127.0.0.1:50668) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=37151 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=37151 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:50668@0x5256e71c-SendThread(127.0.0.1:50668) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=37151 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=37151 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37151 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=37151 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-915468328_17 at /127.0.0.1:45922 [Receiving block BP-1149810363-172.31.14.131-1689192501335:blk_1073741843_1019] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=37151 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: jenkins-hbase4:37151Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37151 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:50668@0x314c4d29 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/362786809.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37151 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=37151 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) - Thread LEAK? -, OpenFileDescriptor=572 (was 577), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=206 (was 233), ProcessCount=170 (was 170), AvailableMemoryMB=7638 (was 7651) 2023-07-12 20:08:47,334 INFO [Listener at localhost/44421] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test2772 Thread=296, OpenFileDescriptor=572, MaxFileDescriptor=60000, SystemLoadAverage=206, ProcessCount=170, AvailableMemoryMB=7636 2023-07-12 20:08:47,334 INFO [Listener at localhost/44421] hbase.LocalHBaseCluster(262): Not alive RS:1;jenkins-hbase4:42213 2023-07-12 20:08:47,335 INFO [Listener at localhost/44421] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,42213,1689192505540 2023-07-12 20:08:47,335 INFO [Listener at localhost/44421] client.TestScannerTimeout(104): START************ test2772 2023-07-12 20:08:47,338 DEBUG [Listener at localhost/44421] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-12 20:08:47,339 DEBUG [Listener at localhost/44421] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:47,339 DEBUG [Listener at localhost/44421] zookeeper.ReadOnlyZKClient(139): Connect 0x32af5d2b to 127.0.0.1:50668 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 20:08:47,347 DEBUG [Listener at localhost/44421] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@179c46a0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 20:08:47,347 ERROR [Listener at localhost/44421] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,43253,1689192505497: die! ***** 2023-07-12 20:08:47,347 ERROR [Listener at localhost/44421] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 20:08:47,347 DEBUG [Listener at localhost/44421] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-12 20:08:47,349 DEBUG [Listener at localhost/44421] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-12 20:08:47,355 DEBUG [Listener at localhost/44421] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-12 20:08:47,357 DEBUG [Listener at localhost/44421] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-12 20:08:47,381 INFO [Listener at localhost/44421] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 965738496, "init": 513802240, "max": 2051014656, "used": 285091096 }, "NonHeapMemoryUsage": { "committed": 122773504, "init": 2555904, "max": -1, "used": 120459040 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 3, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 13022, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 3, "ProcessCallTime_num_ops": 63, "ProcessCallTime_min": 1, "ProcessCallTime_max": 1, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 1, "ProcessCallTime_75th_percentile": 1, "ProcessCallTime_90th_percentile": 1, "ProcessCallTime_95th_percentile": 1, "ProcessCallTime_98th_percentile": 1, "ProcessCallTime_99th_percentile": 1, "ProcessCallTime_99.9th_percentile": 1, "ProcessCallTime_TimeRangeCount_0-1": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 63, "QueueCallTime_min": 0, "QueueCallTime_max": 0, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 0, "QueueCallTime_99th_percentile": 0, "QueueCallTime_99.9th_percentile": 0, "QueueCallTime_TimeRangeCount_0-1": 2, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 63, "TotalCallTime_min": 1, "TotalCallTime_max": 1, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 1, "TotalCallTime_75th_percentile": 1, "TotalCallTime_90th_percentile": 1, "TotalCallTime_95th_percentile": 1, "TotalCallTime_98th_percentile": 1, "TotalCallTime_99th_percentile": 1, "TotalCallTime_99.9th_percentile": 1, "TotalCallTime_TimeRangeCount_0-1": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 63, "ResponseSize_min": 195, "ResponseSize_max": 677, "ResponseSize_mean": 436, "ResponseSize_25th_percentile": 195, "ResponseSize_median": 196, "ResponseSize_75th_percentile": 675, "ResponseSize_90th_percentile": 676, "ResponseSize_95th_percentile": 676, "ResponseSize_98th_percentile": 676, "ResponseSize_99th_percentile": 676, "ResponseSize_99.9th_percentile": 676, "ResponseSize_SizeRangeCount_100-1000": 2, "exceptions.UnknownScannerException": 1, "exceptions": 4, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 0, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 63, "RequestSize_min": 106, "RequestSize_max": 125, "RequestSize_mean": 115, "RequestSize_25th_percentile": 106, "RequestSize_median": 106, "RequestSize_75th_percentile": 124, "RequestSize_90th_percentile": 124, "RequestSize_95th_percentile": 124, "RequestSize_98th_percentile": 124, "RequestSize_99th_percentile": 124, "RequestSize_99.9th_percentile": 124, "RequestSize_SizeRangeCount_100-1000": 2, "sentBytes": 16659 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:50668", "tag.serverName": "jenkins-hbase4.apache.org,43253,1689192505497", "tag.clusterId": "0fd096bf-885b-42ee-9942-e5df6a803c4e", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "regionCount": 3, "storeCount": 5, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 2, "maxStoreFileCount": 1, "memStoreSize": 4186, "storeFileSize": 9852, "maxStoreFileAge": 16776, "minStoreFileAge": 16775, "avgStoreFileAge": 16775, "numReferenceFiles": 0, "regionServerStartTime": 1689192505497, "averageRegionSize": 4679, "storeFileIndexSize": 640, "staticIndexSize": 68, "staticBloomSize": 4, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 1251, "localBytesRead": 1251, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803016, "blockCacheCount": 1, "blockCacheDataBlockCount": 1, "blockCacheSize": 602872, "blockCacheCountHitPercent": 66.66666666666666, "blockCacheExpressHitPercent": 66.66666666666666, "l1CacheSize": 602872, "l1CacheFreeSize": 819803016, "l1CacheCount": 1, "l1CacheEvictionCount": 0, "l1CacheHitCount": 2, "l1CacheMissCount": 1, "l1CacheHitRatio": 0.6666666666666666, "l1CacheMissRatio": 0.3333333333333333, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 3314, "ByteBuffAllocatorPoolAllocationBytes": 1530880, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 41, "totalRowActionRequestCount": 51, "readRequestCount": 37, "filteredReadRequestCount": 0, "writeRequestCount": 14, "rpcGetRequestCount": 6, "rpcFullScanRequestCount": 1, "rpcScanRequestCount": 21, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 14, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 2, "blockCacheHitCountPrimary": 2, "blockCacheHitCachingCount": 2, "blockCacheMissCount": 1, "blockCacheMissCountPrimary": 1, "blockCacheMissCachingCount": 1, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 1, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 2, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 12, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 348, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 20, "ScanTime_min": 0, "ScanTime_max": 1, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 1, "ScanTime_75th_percentile": 1, "ScanTime_90th_percentile": 1, "ScanTime_95th_percentile": 1, "ScanTime_98th_percentile": 1, "ScanTime_99th_percentile": 1, "ScanTime_99.9th_percentile": 1, "ScanTime_TimeRangeCount_0-1": 2, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 14, "Put_min": 0, "Put_max": 0, "Put_mean": 0, "Put_25th_percentile": 0, "Put_median": 0, "Put_75th_percentile": 0, "Put_90th_percentile": 0, "Put_95th_percentile": 0, "Put_98th_percentile": 0, "Put_99th_percentile": 0, "Put_99.9th_percentile": 0, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 2, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 6, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 9852, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 2, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 20, "ScanSize_min": 155, "ScanSize_max": 653, "ScanSize_mean": 404, "ScanSize_25th_percentile": 156, "ScanSize_median": 157, "ScanSize_75th_percentile": 652, "ScanSize_90th_percentile": 652, "ScanSize_95th_percentile": 652, "ScanSize_98th_percentile": 652, "ScanSize_99th_percentile": 652, "ScanSize_99.9th_percentile": 652, "ScanSize_SizeRangeCount_100-1000": 2, "slowGetCount": 0, "flushedMemstoreBytes": 348, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 2, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 54, "ServerReadQueryPerSecond_mean_rate": 2.613604186313324, "ServerReadQueryPerSecond_1min_rate": 7.476487517485488, "ServerReadQueryPerSecond_5min_rate": 9.131802475206852, "ServerReadQueryPerSecond_15min_rate": 9.441325956687528, "ServerWriteQueryPerSecond_count": 26, "ServerWriteQueryPerSecond_mean_rate": 1.2584017571441255, "ServerWriteQueryPerSecond_1min_rate": 4.049764071971306, "ServerWriteQueryPerSecond_5min_rate": 4.946393007403714, "ServerWriteQueryPerSecond_15min_rate": 5.114051559872411 } ] } 2023-07-12 20:08:47,382 WARN [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34965] master.MasterRpcServices(612): jenkins-hbase4.apache.org,43253,1689192505497 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,43253,1689192505497: die! ***** 2023-07-12 20:08:47,384 INFO [Listener at localhost/44421] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,43253,1689192505497' ***** 2023-07-12 20:08:47,384 INFO [Listener at localhost/44421] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-12 20:08:47,385 INFO [RS:0;jenkins-hbase4:43253] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 20:08:47,385 INFO [RS:0;jenkins-hbase4:43253] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-12 20:08:47,385 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 20:08:47,385 INFO [RS:0;jenkins-hbase4:43253] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-12 20:08:47,385 INFO [RS:0;jenkins-hbase4:43253] regionserver.HRegionServer(3305): Received CLOSE for c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:47,385 INFO [RS:0;jenkins-hbase4:43253] regionserver.HRegionServer(3305): Received CLOSE for 4d090c063c934b8812e620114898ad9b 2023-07-12 20:08:47,385 INFO [RS:0;jenkins-hbase4:43253] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,43253,1689192505497 2023-07-12 20:08:47,385 DEBUG [RS:0;jenkins-hbase4:43253] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c6607c3 to 127.0.0.1:50668 2023-07-12 20:08:47,385 DEBUG [RS:0;jenkins-hbase4:43253] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 20:08:47,386 INFO [RS:0;jenkins-hbase4:43253] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 20:08:47,386 INFO [RS:0;jenkins-hbase4:43253] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 20:08:47,386 INFO [RS:0;jenkins-hbase4:43253] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 20:08:47,386 INFO [RS:0;jenkins-hbase4:43253] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-12 20:08:47,386 INFO [RS:0;jenkins-hbase4:43253] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-12 20:08:47,386 DEBUG [RS:0;jenkins-hbase4:43253] regionserver.HRegionServer(1478): Online Regions={c0bf5c648500555868549c4d8dbe5724=hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724., 4d090c063c934b8812e620114898ad9b=t,,1689192508262.4d090c063c934b8812e620114898ad9b., 1588230740=hbase:meta,,1.1588230740} 2023-07-12 20:08:47,386 DEBUG [RS:0;jenkins-hbase4:43253] regionserver.HRegionServer(1504): Waiting on 1588230740, 4d090c063c934b8812e620114898ad9b, c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:47,387 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c0bf5c648500555868549c4d8dbe5724, disabling compactions & flushes 2023-07-12 20:08:47,387 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 20:08:47,387 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 20:08:47,387 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:47,387 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 20:08:47,387 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:47,387 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. after waiting 0 ms 2023-07-12 20:08:47,387 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 20:08:47,387 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:47,387 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 20:08:47,396 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 4186 in region hbase:meta,,1.1588230740 2023-07-12 20:08:47,396 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 20:08:47,403 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 20:08:47,403 DEBUG [hconnection-0x28a14a7c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 20:08:47,403 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:47,403 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 20:08:47,407 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-12 20:08:47,408 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c0bf5c648500555868549c4d8dbe5724: 2023-07-12 20:08:47,408 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:47,415 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 4d090c063c934b8812e620114898ad9b, disabling compactions & flushes 2023-07-12 20:08:47,415 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:47,415 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:47,415 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689192508262.4d090c063c934b8812e620114898ad9b. after waiting 0 ms 2023-07-12 20:08:47,415 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:47,422 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:47,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 4d090c063c934b8812e620114898ad9b: 2023-07-12 20:08:47,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:47,424 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58094, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 20:08:47,424 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=43253] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,43253,1689192505497 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 20:08:47,424 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=43253] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:58094 deadline: 1689192587424, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,43253,1689192505497 aborting 2023-07-12 20:08:47,532 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=43253] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,43253,1689192505497 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 20:08:47,532 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=43253] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:58094 deadline: 1689192587531, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,43253,1689192505497 aborting 2023-07-12 20:08:47,532 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 20:08:47,586 INFO [RS:0;jenkins-hbase4:43253] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,43253,1689192505497; all regions closed. 2023-07-12 20:08:47,612 DEBUG [RS:0;jenkins-hbase4:43253] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 20:08:47,612 INFO [RS:0;jenkins-hbase4:43253] regionserver.LeaseManager(133): Closed leases 2023-07-12 20:08:47,612 INFO [RS:0;jenkins-hbase4:43253] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-12 20:08:47,613 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 20:08:47,614 INFO [RS:0;jenkins-hbase4:43253] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:43253 2023-07-12 20:08:47,623 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): regionserver:43253-0x1015b269bb50001, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43253,1689192505497 2023-07-12 20:08:47,623 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 20:08:47,623 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): regionserver:37151-0x1015b269bb50008, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43253,1689192505497 2023-07-12 20:08:47,623 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): regionserver:37151-0x1015b269bb50008, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 20:08:47,623 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): regionserver:43253-0x1015b269bb50001, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 20:08:47,625 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,43253,1689192505497] 2023-07-12 20:08:47,625 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,43253,1689192505497; numProcessing=1 2023-07-12 20:08:47,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37151-0x1015b269bb50008, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37151,1689192510197 2023-07-12 20:08:47,626 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,43253,1689192505497 znode expired, triggering replicatorRemoved event 2023-07-12 20:08:47,628 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,43253,1689192505497 already deleted, retry=false 2023-07-12 20:08:47,628 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,43253,1689192505497 on jenkins-hbase4.apache.org,34965,1689192504325 2023-07-12 20:08:47,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37151-0x1015b269bb50008, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37151,1689192510197 2023-07-12 20:08:47,629 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,43253,1689192505497, splitWal=true, meta=true 2023-07-12 20:08:47,629 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=19 for jenkins-hbase4.apache.org,43253,1689192505497 (carryingMeta=true) jenkins-hbase4.apache.org,43253,1689192505497/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@193b1439[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-12 20:08:47,630 INFO [PEWorker-1] procedure.ServerCrashProcedure(161): Start pid=19, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,43253,1689192505497, splitWal=true, meta=true 2023-07-12 20:08:47,632 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=19, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,43253,1689192505497, splitWal=true, meta=true, isMeta: true 2023-07-12 20:08:47,635 DEBUG [PEWorker-1] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting 2023-07-12 20:08:47,637 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,43253,1689192505497 WAL count=1, meta=true 2023-07-12 20:08:47,637 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C43253%2C1689192505497.meta.1689192507124.meta}] 2023-07-12 20:08:47,639 DEBUG [PEWorker-4] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,37151,1689192510197 2023-07-12 20:08:47,640 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C43253%2C1689192505497.meta.1689192507124.meta, worker=jenkins-hbase4.apache.org,37151,1689192510197}] 2023-07-12 20:08:47,725 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): regionserver:43253-0x1015b269bb50001, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 20:08:47,725 INFO [RS:0;jenkins-hbase4:43253] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,43253,1689192505497; zookeeper connection closed. 2023-07-12 20:08:47,725 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): regionserver:43253-0x1015b269bb50001, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 20:08:47,726 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@40e644b0] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@40e644b0 2023-07-12 20:08:47,736 WARN [RPCClient-NioEventLoopGroup-5-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:43253 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:43253 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 20:08:47,738 DEBUG [RPCClient-NioEventLoopGroup-5-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:43253 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:43253 2023-07-12 20:08:47,793 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,37151,1689192510197 2023-07-12 20:08:47,793 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 20:08:47,795 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59662, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 20:08:47,795 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=37151] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=21 2023-07-12 20:08:47,810 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting/jenkins-hbase4.apache.org%2C43253%2C1689192505497.meta.1689192507124.meta, size=5.1 K (5185bytes) 2023-07-12 20:08:47,810 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting/jenkins-hbase4.apache.org%2C43253%2C1689192505497.meta.1689192507124.meta 2023-07-12 20:08:47,811 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting/jenkins-hbase4.apache.org%2C43253%2C1689192505497.meta.1689192507124.meta after 1ms 2023-07-12 20:08:47,816 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting/jenkins-hbase4.apache.org%2C43253%2C1689192505497.meta.1689192507124.meta took 6ms 2023-07-12 20:08:47,821 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-12 20:08:47,823 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting/jenkins-hbase4.apache.org%2C43253%2C1689192505497.meta.1689192507124.meta so closing down 2023-07-12 20:08:47,823 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-12 20:08:47,826 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43253%2C1689192505497.meta.1689192507124.meta.temp 2023-07-12 20:08:47,827 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-12 20:08:47,842 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43253%2C1689192505497.meta.1689192507124.meta.temp (wrote 14 edits, skipped 0 edits in 0 ms) 2023-07-12 20:08:47,844 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43253%2C1689192505497.meta.1689192507124.meta.temp to hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-12 20:08:47,844 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 15 edits across 1 Regions in 27 ms; skipped=1; WAL=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting/jenkins-hbase4.apache.org%2C43253%2C1689192505497.meta.1689192507124.meta, size=5.1 K, length=5185, corrupted=false, cancelled=false 2023-07-12 20:08:47,844 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting/jenkins-hbase4.apache.org%2C43253%2C1689192505497.meta.1689192507124.meta, journal: Splitting hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting/jenkins-hbase4.apache.org%2C43253%2C1689192505497.meta.1689192507124.meta, size=5.1 K (5185bytes) at 1689192527810Finishing writing output for hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting/jenkins-hbase4.apache.org%2C43253%2C1689192505497.meta.1689192507124.meta so closing down at 1689192527823 (+13 ms)Creating recovered edits writer path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43253%2C1689192505497.meta.1689192507124.meta.temp at 1689192527827 (+4 ms)3 split writer threads finished at 1689192527827Closed recovered edits writer path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43253%2C1689192505497.meta.1689192507124.meta.temp (wrote 14 edits, skipped 0 edits in 0 ms) at 1689192527842 (+15 ms)Rename recovered edits hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43253%2C1689192505497.meta.1689192507124.meta.temp to hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 at 1689192527844 (+2 ms)Processed 15 edits across 1 Regions in 27 ms; skipped=1; WAL=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting/jenkins-hbase4.apache.org%2C43253%2C1689192505497.meta.1689192507124.meta, size=5.1 K, length=5185, corrupted=false, cancelled=false at 1689192527844 2023-07-12 20:08:47,844 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting/jenkins-hbase4.apache.org%2C43253%2C1689192505497.meta.1689192507124.meta 2023-07-12 20:08:47,844 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=21 2023-07-12 20:08:47,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34965] master.HMaster(3641): Remote procedure done, pid=21 2023-07-12 20:08:47,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34965] wal.WALSplitUtil(141): Moved hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting/jenkins-hbase4.apache.org%2C43253%2C1689192505497.meta.1689192507124.meta to hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/oldWALs 2023-07-12 20:08:47,852 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-07-12 20:08:47,852 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C43253%2C1689192505497.meta.1689192507124.meta, worker=jenkins-hbase4.apache.org,37151,1689192510197 in 209 msec 2023-07-12 20:08:47,853 DEBUG [PEWorker-5] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,37151,1689192510197 2023-07-12 20:08:47,855 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-07-12 20:08:47,856 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C43253%2C1689192505497.meta.1689192507124.meta, worker=jenkins-hbase4.apache.org,37151,1689192510197 in 216 msec 2023-07-12 20:08:47,857 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting dir is empty, no logs to split. 2023-07-12 20:08:47,858 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,43253,1689192505497 WAL count=0, meta=true 2023-07-12 20:08:47,858 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,43253,1689192505497 WAL splitting is done? wals=0, meta=true 2023-07-12 20:08:47,859 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-12 20:08:47,860 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=22, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-12 20:08:47,861 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=22, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-12 20:08:48,011 DEBUG [jenkins-hbase4:34965] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-12 20:08:48,011 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,37151,1689192510197, state=OPENING 2023-07-12 20:08:48,013 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 20:08:48,013 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 20:08:48,013 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,37151,1689192510197}] 2023-07-12 20:08:48,039 WARN [ReadOnlyZKClient-127.0.0.1:50668@0x32af5d2b] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-12 20:08:48,039 DEBUG [hconnection-0x28a14a7c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 20:08:48,042 INFO [RS-EventLoopGroup-6-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59668, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 20:08:48,043 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=37151] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:59668 deadline: 1689192588042, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,37151,1689192510197 2023-07-12 20:08:48,172 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-12 20:08:48,172 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 20:08:48,175 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C37151%2C1689192510197.meta, suffix=.meta, logDir=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,37151,1689192510197, archiveDir=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/oldWALs, maxLogs=32 2023-07-12 20:08:48,193 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43453,DS-c995922d-4442-40c9-9e1a-e075f395e478,DISK] 2023-07-12 20:08:48,194 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41651,DS-8370ea8b-9cf9-4e9e-a7c3-2af949363208,DISK] 2023-07-12 20:08:48,197 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,37151,1689192510197/jenkins-hbase4.apache.org%2C37151%2C1689192510197.meta.1689192528177.meta 2023-07-12 20:08:48,202 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43453,DS-c995922d-4442-40c9-9e1a-e075f395e478,DISK], DatanodeInfoWithStorage[127.0.0.1:41651,DS-8370ea8b-9cf9-4e9e-a7c3-2af949363208,DISK]] 2023-07-12 20:08:48,203 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-12 20:08:48,203 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-12 20:08:48,203 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-12 20:08:48,204 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-12 20:08:48,204 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-12 20:08:48,204 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 20:08:48,204 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-12 20:08:48,204 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-12 20:08:48,206 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 20:08:48,207 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/info 2023-07-12 20:08:48,207 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/info 2023-07-12 20:08:48,208 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 20:08:48,208 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 20:08:48,208 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 20:08:48,210 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/rep_barrier 2023-07-12 20:08:48,210 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/rep_barrier 2023-07-12 20:08:48,210 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 20:08:48,211 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 20:08:48,211 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 20:08:48,213 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/table 2023-07-12 20:08:48,213 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/table 2023-07-12 20:08:48,213 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 20:08:48,214 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 20:08:48,215 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740 2023-07-12 20:08:48,217 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740 2023-07-12 20:08:48,218 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-12 20:08:48,225 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 40, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=17, path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-12 20:08:48,225 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=4.09 KB heapSize=7.79 KB 2023-07-12 20:08:48,242 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.93 KB at sequenceid=17 (bloomFilter=false), to=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/.tmp/info/79e08a3525f64740acf770ebd13e0e33 2023-07-12 20:08:48,263 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=160 B at sequenceid=17 (bloomFilter=false), to=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/.tmp/table/a50e187ee91d4b0ea007f6ed4cbe528b 2023-07-12 20:08:48,270 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/.tmp/info/79e08a3525f64740acf770ebd13e0e33 as hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/info/79e08a3525f64740acf770ebd13e0e33 2023-07-12 20:08:48,276 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/info/79e08a3525f64740acf770ebd13e0e33, entries=36, sequenceid=17, filesize=8.6 K 2023-07-12 20:08:48,278 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/.tmp/table/a50e187ee91d4b0ea007f6ed4cbe528b as hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/table/a50e187ee91d4b0ea007f6ed4cbe528b 2023-07-12 20:08:48,285 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/table/a50e187ee91d4b0ea007f6ed4cbe528b, entries=4, sequenceid=17, filesize=4.7 K 2023-07-12 20:08:48,286 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~4.09 KB/4186, heapSize ~7.51 KB/7688, currentSize=0 B/0 for 1588230740 in 61ms, sequenceid=17, compaction requested=false; wal=null 2023-07-12 20:08:48,287 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-12 20:08:48,290 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 20:08:48,291 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 20:08:48,294 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=1 2023-07-12 20:08:48,295 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=18; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73281451, jitterRate=0.09197871387004852}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 20:08:48,295 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 20:08:48,296 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=23, masterSystemTime=1689192528166 2023-07-12 20:08:48,298 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-12 20:08:48,299 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-12 20:08:48,300 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,37151,1689192510197, state=OPEN 2023-07-12 20:08:48,301 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 20:08:48,301 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 20:08:48,304 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-12 20:08:48,304 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,37151,1689192510197 in 288 msec 2023-07-12 20:08:48,307 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=19 2023-07-12 20:08:48,307 INFO [PEWorker-4] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,43253,1689192505497 had 3 regions 2023-07-12 20:08:48,307 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 445 msec 2023-07-12 20:08:48,308 INFO [PEWorker-4] procedure.ServerCrashProcedure(300): Splitting WALs pid=19, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,43253,1689192505497, splitWal=true, meta=true, isMeta: false 2023-07-12 20:08:48,311 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,43253,1689192505497 WAL count=1, meta=false 2023-07-12 20:08:48,311 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=19, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C43253%2C1689192505497.1689192506894}] 2023-07-12 20:08:48,313 DEBUG [PEWorker-3] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,37151,1689192510197 2023-07-12 20:08:48,314 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C43253%2C1689192505497.1689192506894, worker=jenkins-hbase4.apache.org,37151,1689192510197}] 2023-07-12 20:08:48,467 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=37151] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=25 2023-07-12 20:08:48,479 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting/jenkins-hbase4.apache.org%2C43253%2C1689192505497.1689192506894, size=721 (721bytes) 2023-07-12 20:08:48,480 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting/jenkins-hbase4.apache.org%2C43253%2C1689192505497.1689192506894 2023-07-12 20:08:48,480 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting/jenkins-hbase4.apache.org%2C43253%2C1689192505497.1689192506894 after 0ms 2023-07-12 20:08:48,484 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting/jenkins-hbase4.apache.org%2C43253%2C1689192505497.1689192506894 took 5ms 2023-07-12 20:08:48,487 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 4d090c063c934b8812e620114898ad9b: last_flushed_sequence_id: 13 store_sequence_id { family_name: "f" sequence_id: 13 } 2023-07-12 20:08:48,488 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for c0bf5c648500555868549c4d8dbe5724: last_flushed_sequence_id: 5 store_sequence_id { family_name: "info" sequence_id: 5 } 2023-07-12 20:08:48,488 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting/jenkins-hbase4.apache.org%2C43253%2C1689192505497.1689192506894 so closing down 2023-07-12 20:08:48,488 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-12 20:08:48,490 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-12 20:08:48,490 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 2 edits across 0 Regions in 6 ms; skipped=2; WAL=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting/jenkins-hbase4.apache.org%2C43253%2C1689192505497.1689192506894, size=721, length=721, corrupted=false, cancelled=false 2023-07-12 20:08:48,490 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting/jenkins-hbase4.apache.org%2C43253%2C1689192505497.1689192506894, journal: Splitting hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting/jenkins-hbase4.apache.org%2C43253%2C1689192505497.1689192506894, size=721 (721bytes) at 1689192528480Finishing writing output for hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting/jenkins-hbase4.apache.org%2C43253%2C1689192505497.1689192506894 so closing down at 1689192528488 (+8 ms)3 split writer threads finished at 1689192528490 (+2 ms)Processed 2 edits across 0 Regions in 6 ms; skipped=2; WAL=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting/jenkins-hbase4.apache.org%2C43253%2C1689192505497.1689192506894, size=721, length=721, corrupted=false, cancelled=false at 1689192528490 2023-07-12 20:08:48,490 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting/jenkins-hbase4.apache.org%2C43253%2C1689192505497.1689192506894 2023-07-12 20:08:48,490 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=25 2023-07-12 20:08:48,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34965] master.HMaster(3641): Remote procedure done, pid=25 2023-07-12 20:08:48,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34965] wal.WALSplitUtil(141): Moved hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting/jenkins-hbase4.apache.org%2C43253%2C1689192505497.1689192506894 to hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/oldWALs 2023-07-12 20:08:48,497 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-07-12 20:08:48,497 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C43253%2C1689192505497.1689192506894, worker=jenkins-hbase4.apache.org,37151,1689192510197 in 180 msec 2023-07-12 20:08:48,498 DEBUG [PEWorker-1] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,37151,1689192510197 2023-07-12 20:08:48,501 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=19 2023-07-12 20:08:48,501 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=19, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C43253%2C1689192505497.1689192506894, worker=jenkins-hbase4.apache.org,37151,1689192510197 in 187 msec 2023-07-12 20:08:48,503 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/WALs/jenkins-hbase4.apache.org,43253,1689192505497-splitting dir is empty, no logs to split. 2023-07-12 20:08:48,503 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,43253,1689192505497 WAL count=0, meta=false 2023-07-12 20:08:48,503 DEBUG [PEWorker-4] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,43253,1689192505497 WAL splitting is done? wals=0, meta=false 2023-07-12 20:08:48,505 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=4d090c063c934b8812e620114898ad9b, ASSIGN}, {pid=27, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c0bf5c648500555868549c4d8dbe5724, ASSIGN}] 2023-07-12 20:08:48,507 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=27, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c0bf5c648500555868549c4d8dbe5724, ASSIGN 2023-07-12 20:08:48,507 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=4d090c063c934b8812e620114898ad9b, ASSIGN 2023-07-12 20:08:48,508 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=27, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=c0bf5c648500555868549c4d8dbe5724, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-12 20:08:48,508 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=26, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=4d090c063c934b8812e620114898ad9b, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-12 20:08:48,552 DEBUG [hconnection-0x28a14a7c-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:43253 this server is in the failed servers list 2023-07-12 20:08:48,659 DEBUG [jenkins-hbase4:34965] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-12 20:08:48,659 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=c0bf5c648500555868549c4d8dbe5724, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37151,1689192510197 2023-07-12 20:08:48,659 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=4d090c063c934b8812e620114898ad9b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37151,1689192510197 2023-07-12 20:08:48,659 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689192528659"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689192528659"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689192528659"}]},"ts":"1689192528659"} 2023-07-12 20:08:48,659 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689192508262.4d090c063c934b8812e620114898ad9b.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689192528659"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689192528659"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689192528659"}]},"ts":"1689192528659"} 2023-07-12 20:08:48,660 WARN [RS-EventLoopGroup-4-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:43253 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:43253 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 20:08:48,662 DEBUG [RS-EventLoopGroup-4-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:43253 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:43253 2023-07-12 20:08:48,662 DEBUG [hconnection-0x28a14a7c-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:43253 this server is in the failed servers list 2023-07-12 20:08:48,763 DEBUG [PEWorker-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 20:08:48,765 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59676, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 20:08:48,768 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=26, state=RUNNABLE; OpenRegionProcedure 4d090c063c934b8812e620114898ad9b, server=jenkins-hbase4.apache.org,37151,1689192510197}] 2023-07-12 20:08:48,769 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=27, state=RUNNABLE; OpenRegionProcedure c0bf5c648500555868549c4d8dbe5724, server=jenkins-hbase4.apache.org,37151,1689192510197}] 2023-07-12 20:08:48,870 DEBUG [hconnection-0x28a14a7c-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:43253 this server is in the failed servers list 2023-07-12 20:08:48,925 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:48,925 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c0bf5c648500555868549c4d8dbe5724, NAME => 'hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724.', STARTKEY => '', ENDKEY => ''} 2023-07-12 20:08:48,926 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:48,926 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 20:08:48,926 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:48,926 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:48,927 INFO [StoreOpener-c0bf5c648500555868549c4d8dbe5724-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:48,928 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:48,929 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4d090c063c934b8812e620114898ad9b, NAME => 't,,1689192508262.4d090c063c934b8812e620114898ad9b.', STARTKEY => '', ENDKEY => ''} 2023-07-12 20:08:48,929 DEBUG [StoreOpener-c0bf5c648500555868549c4d8dbe5724-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724/info 2023-07-12 20:08:48,929 DEBUG [StoreOpener-c0bf5c648500555868549c4d8dbe5724-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724/info 2023-07-12 20:08:48,929 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 4d090c063c934b8812e620114898ad9b 2023-07-12 20:08:48,929 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689192508262.4d090c063c934b8812e620114898ad9b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 20:08:48,929 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 4d090c063c934b8812e620114898ad9b 2023-07-12 20:08:48,929 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 4d090c063c934b8812e620114898ad9b 2023-07-12 20:08:48,929 INFO [StoreOpener-c0bf5c648500555868549c4d8dbe5724-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c0bf5c648500555868549c4d8dbe5724 columnFamilyName info 2023-07-12 20:08:48,931 INFO [StoreOpener-4d090c063c934b8812e620114898ad9b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 4d090c063c934b8812e620114898ad9b 2023-07-12 20:08:48,932 DEBUG [StoreOpener-4d090c063c934b8812e620114898ad9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b/f 2023-07-12 20:08:48,932 DEBUG [StoreOpener-4d090c063c934b8812e620114898ad9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b/f 2023-07-12 20:08:48,933 INFO [StoreOpener-4d090c063c934b8812e620114898ad9b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4d090c063c934b8812e620114898ad9b columnFamilyName f 2023-07-12 20:08:48,939 DEBUG [StoreOpener-c0bf5c648500555868549c4d8dbe5724-1] regionserver.HStore(539): loaded hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724/info/918473fcd52d467483dc49baa2aece6d 2023-07-12 20:08:48,940 INFO [StoreOpener-c0bf5c648500555868549c4d8dbe5724-1] regionserver.HStore(310): Store=c0bf5c648500555868549c4d8dbe5724/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 20:08:48,941 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:48,943 DEBUG [StoreOpener-4d090c063c934b8812e620114898ad9b-1] regionserver.HStore(539): loaded hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b/f/fadadbb9f85b49099db7f2fa514cdd64 2023-07-12 20:08:48,943 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:48,943 INFO [StoreOpener-4d090c063c934b8812e620114898ad9b-1] regionserver.HStore(310): Store=4d090c063c934b8812e620114898ad9b/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 20:08:48,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b 2023-07-12 20:08:48,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b 2023-07-12 20:08:48,947 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:48,950 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724/recovered.edits/6.seqid, newMaxSeqId=6, maxSeqId=5 2023-07-12 20:08:48,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 4d090c063c934b8812e620114898ad9b 2023-07-12 20:08:48,951 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c0bf5c648500555868549c4d8dbe5724; next sequenceid=7; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75383615, jitterRate=0.12330339848995209}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 20:08:48,951 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c0bf5c648500555868549c4d8dbe5724: 2023-07-12 20:08:48,953 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724., pid=29, masterSystemTime=1689192528920 2023-07-12 20:08:48,953 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b/recovered.edits/14.seqid, newMaxSeqId=14, maxSeqId=13 2023-07-12 20:08:48,955 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 4d090c063c934b8812e620114898ad9b; next sequenceid=15; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72882820, jitterRate=0.08603864908218384}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 20:08:48,955 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 4d090c063c934b8812e620114898ad9b: 2023-07-12 20:08:48,955 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:48,955 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:48,956 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689192508262.4d090c063c934b8812e620114898ad9b., pid=28, masterSystemTime=1689192528920 2023-07-12 20:08:48,956 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=c0bf5c648500555868549c4d8dbe5724, regionState=OPEN, openSeqNum=7, regionLocation=jenkins-hbase4.apache.org,37151,1689192510197 2023-07-12 20:08:48,956 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689192528956"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689192528956"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689192528956"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689192528956"}]},"ts":"1689192528956"} 2023-07-12 20:08:48,957 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:48,957 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:48,958 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=4d090c063c934b8812e620114898ad9b, regionState=OPEN, openSeqNum=15, regionLocation=jenkins-hbase4.apache.org,37151,1689192510197 2023-07-12 20:08:48,959 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689192508262.4d090c063c934b8812e620114898ad9b.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689192528958"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689192528958"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689192528958"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689192528958"}]},"ts":"1689192528958"} 2023-07-12 20:08:48,961 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=27 2023-07-12 20:08:48,961 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=27, state=SUCCESS; OpenRegionProcedure c0bf5c648500555868549c4d8dbe5724, server=jenkins-hbase4.apache.org,37151,1689192510197 in 190 msec 2023-07-12 20:08:48,963 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=c0bf5c648500555868549c4d8dbe5724, ASSIGN in 456 msec 2023-07-12 20:08:48,964 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=28, resume processing ppid=26 2023-07-12 20:08:48,964 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=26, state=SUCCESS; OpenRegionProcedure 4d090c063c934b8812e620114898ad9b, server=jenkins-hbase4.apache.org,37151,1689192510197 in 193 msec 2023-07-12 20:08:48,966 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=19 2023-07-12 20:08:48,966 INFO [PEWorker-3] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,43253,1689192505497 after splitting done 2023-07-12 20:08:48,966 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=t, region=4d090c063c934b8812e620114898ad9b, ASSIGN in 459 msec 2023-07-12 20:08:48,966 DEBUG [PEWorker-3] master.DeadServer(114): Removed jenkins-hbase4.apache.org,43253,1689192505497 from processing; numProcessing=0 2023-07-12 20:08:48,968 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,43253,1689192505497, splitWal=true, meta=true in 1.3380 sec 2023-07-12 20:08:49,183 DEBUG [Listener at localhost/44421] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x32af5d2b to 127.0.0.1:50668 2023-07-12 20:08:49,183 DEBUG [Listener at localhost/44421] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 20:08:49,183 INFO [Listener at localhost/44421] client.TestScannerTimeout(124): END ************ test2772 2023-07-12 20:08:49,196 INFO [Listener at localhost/44421] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test2772 Thread=278 (was 296), OpenFileDescriptor=577 (was 572) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=206 (was 206), ProcessCount=170 (was 170), AvailableMemoryMB=7629 (was 7636) 2023-07-12 20:08:49,196 INFO [Listener at localhost/44421] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-12 20:08:49,197 INFO [Listener at localhost/44421] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-12 20:08:49,197 DEBUG [Listener at localhost/44421] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2163cf9f to 127.0.0.1:50668 2023-07-12 20:08:49,197 DEBUG [Listener at localhost/44421] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 20:08:49,197 DEBUG [Listener at localhost/44421] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-12 20:08:49,197 DEBUG [Listener at localhost/44421] util.JVMClusterUtil(257): Found active master hash=1998443920, stopped=false 2023-07-12 20:08:49,197 INFO [Listener at localhost/44421] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,34965,1689192504325 2023-07-12 20:08:49,204 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 20:08:49,204 INFO [Listener at localhost/44421] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 20:08:49,204 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 20:08:49,204 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): regionserver:37151-0x1015b269bb50008, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 20:08:49,205 DEBUG [Listener at localhost/44421] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x16d66d79 to 127.0.0.1:50668 2023-07-12 20:08:49,205 DEBUG [Listener at localhost/44421] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 20:08:49,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 20:08:49,205 INFO [Listener at localhost/44421] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,37151,1689192510197' ***** 2023-07-12 20:08:49,205 INFO [Listener at localhost/44421] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-12 20:08:49,205 INFO [RS:2;jenkins-hbase4:37151] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 20:08:49,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37151-0x1015b269bb50008, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 20:08:49,205 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 20:08:49,205 INFO [RS:2;jenkins-hbase4:37151] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-12 20:08:49,206 INFO [RS:2;jenkins-hbase4:37151] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-12 20:08:49,206 INFO [RS:2;jenkins-hbase4:37151] regionserver.HRegionServer(3305): Received CLOSE for c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:49,206 INFO [RS:2;jenkins-hbase4:37151] regionserver.HRegionServer(3305): Received CLOSE for 4d090c063c934b8812e620114898ad9b 2023-07-12 20:08:49,206 INFO [RS:2;jenkins-hbase4:37151] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,37151,1689192510197 2023-07-12 20:08:49,206 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c0bf5c648500555868549c4d8dbe5724, disabling compactions & flushes 2023-07-12 20:08:49,206 DEBUG [RS:2;jenkins-hbase4:37151] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x314c4d29 to 127.0.0.1:50668 2023-07-12 20:08:49,207 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:49,207 DEBUG [RS:2;jenkins-hbase4:37151] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 20:08:49,207 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:49,207 INFO [RS:2;jenkins-hbase4:37151] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 20:08:49,207 INFO [RS:2;jenkins-hbase4:37151] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 20:08:49,207 INFO [RS:2;jenkins-hbase4:37151] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 20:08:49,207 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. after waiting 0 ms 2023-07-12 20:08:49,207 INFO [RS:2;jenkins-hbase4:37151] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-12 20:08:49,207 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:49,207 INFO [RS:2;jenkins-hbase4:37151] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-12 20:08:49,207 DEBUG [RS:2;jenkins-hbase4:37151] regionserver.HRegionServer(1478): Online Regions={c0bf5c648500555868549c4d8dbe5724=hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724., 4d090c063c934b8812e620114898ad9b=t,,1689192508262.4d090c063c934b8812e620114898ad9b., 1588230740=hbase:meta,,1.1588230740} 2023-07-12 20:08:49,208 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 20:08:49,208 DEBUG [RS:2;jenkins-hbase4:37151] regionserver.HRegionServer(1504): Waiting on 1588230740, 4d090c063c934b8812e620114898ad9b, c0bf5c648500555868549c4d8dbe5724 2023-07-12 20:08:49,208 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 20:08:49,208 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 20:08:49,208 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 20:08:49,208 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 20:08:49,208 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-12 20:08:49,213 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/namespace/c0bf5c648500555868549c4d8dbe5724/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=6 2023-07-12 20:08:49,214 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:49,214 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c0bf5c648500555868549c4d8dbe5724: 2023-07-12 20:08:49,214 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689192507416.c0bf5c648500555868549c4d8dbe5724. 2023-07-12 20:08:49,215 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 4d090c063c934b8812e620114898ad9b, disabling compactions & flushes 2023-07-12 20:08:49,215 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:49,215 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:49,215 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689192508262.4d090c063c934b8812e620114898ad9b. after waiting 0 ms 2023-07-12 20:08:49,215 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:49,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/default/t/4d090c063c934b8812e620114898ad9b/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=14 2023-07-12 20:08:49,222 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:49,222 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 4d090c063c934b8812e620114898ad9b: 2023-07-12 20:08:49,222 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689192508262.4d090c063c934b8812e620114898ad9b. 2023-07-12 20:08:49,239 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=24 (bloomFilter=false), to=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/.tmp/info/919dc131d5984282bc3006ded93209c5 2023-07-12 20:08:49,246 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/.tmp/info/919dc131d5984282bc3006ded93209c5 as hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/info/919dc131d5984282bc3006ded93209c5 2023-07-12 20:08:49,253 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/info/919dc131d5984282bc3006ded93209c5, entries=16, sequenceid=24, filesize=6.4 K 2023-07-12 20:08:49,254 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1792, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 46ms, sequenceid=24, compaction requested=false 2023-07-12 20:08:49,255 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 20:08:49,258 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-12 20:08:49,259 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-12 20:08:49,261 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/data/hbase/meta/1588230740/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=17 2023-07-12 20:08:49,262 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 20:08:49,262 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 20:08:49,262 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 20:08:49,262 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-12 20:08:49,408 INFO [RS:2;jenkins-hbase4:37151] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,37151,1689192510197; all regions closed. 2023-07-12 20:08:49,414 DEBUG [RS:2;jenkins-hbase4:37151] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/oldWALs 2023-07-12 20:08:49,414 INFO [RS:2;jenkins-hbase4:37151] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C37151%2C1689192510197.meta:.meta(num 1689192528177) 2023-07-12 20:08:49,419 DEBUG [RS:2;jenkins-hbase4:37151] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/oldWALs 2023-07-12 20:08:49,419 INFO [RS:2;jenkins-hbase4:37151] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C37151%2C1689192510197:(num 1689192510383) 2023-07-12 20:08:49,419 DEBUG [RS:2;jenkins-hbase4:37151] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 20:08:49,419 INFO [RS:2;jenkins-hbase4:37151] regionserver.LeaseManager(133): Closed leases 2023-07-12 20:08:49,420 INFO [RS:2;jenkins-hbase4:37151] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-12 20:08:49,420 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 20:08:49,420 INFO [RS:2;jenkins-hbase4:37151] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:37151 2023-07-12 20:08:49,422 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): regionserver:37151-0x1015b269bb50008, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,37151,1689192510197 2023-07-12 20:08:49,422 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 20:08:49,422 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): regionserver:37151-0x1015b269bb50008, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 20:08:49,423 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,37151,1689192510197] 2023-07-12 20:08:49,423 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,37151,1689192510197; numProcessing=1 2023-07-12 20:08:49,426 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,37151,1689192510197 already deleted, retry=false 2023-07-12 20:08:49,426 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,37151,1689192510197 expired; onlineServers=0 2023-07-12 20:08:49,426 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,34965,1689192504325' ***** 2023-07-12 20:08:49,426 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-12 20:08:49,426 DEBUG [M:0;jenkins-hbase4:34965] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3620dfab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-12 20:08:49,426 INFO [M:0;jenkins-hbase4:34965] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,34965,1689192504325 2023-07-12 20:08:49,426 INFO [M:0;jenkins-hbase4:34965] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,34965,1689192504325; all regions closed. 2023-07-12 20:08:49,426 DEBUG [M:0;jenkins-hbase4:34965] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 20:08:49,426 DEBUG [M:0;jenkins-hbase4:34965] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 20:08:49,427 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 20:08:49,427 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689192506599] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689192506599,5,FailOnTimeoutGroup] 2023-07-12 20:08:49,427 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689192506600] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689192506600,5,FailOnTimeoutGroup] 2023-07-12 20:08:49,427 DEBUG [M:0;jenkins-hbase4:34965] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 20:08:49,427 INFO [M:0;jenkins-hbase4:34965] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 20:08:49,427 INFO [M:0;jenkins-hbase4:34965] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 20:08:49,428 INFO [M:0;jenkins-hbase4:34965] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-12 20:08:49,428 DEBUG [M:0;jenkins-hbase4:34965] master.HMaster(1512): Stopping service threads 2023-07-12 20:08:49,428 INFO [M:0;jenkins-hbase4:34965] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 20:08:49,428 ERROR [M:0;jenkins-hbase4:34965] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[IPC Client (298043509) connection to localhost/127.0.0.1:45739 from jenkins,5,PEWorkerGroup] 2023-07-12 20:08:49,428 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 20:08:49,428 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 20:08:49,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 20:08:49,428 INFO [M:0;jenkins-hbase4:34965] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 20:08:49,429 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 20:08:49,429 DEBUG [M:0;jenkins-hbase4:34965] zookeeper.ZKUtil(398): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 20:08:49,429 WARN [M:0;jenkins-hbase4:34965] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 20:08:49,429 INFO [M:0;jenkins-hbase4:34965] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 20:08:49,429 INFO [M:0;jenkins-hbase4:34965] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-12 20:08:49,430 DEBUG [M:0;jenkins-hbase4:34965] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 20:08:49,430 INFO [M:0;jenkins-hbase4:34965] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 20:08:49,430 DEBUG [M:0;jenkins-hbase4:34965] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 20:08:49,430 DEBUG [M:0;jenkins-hbase4:34965] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 20:08:49,430 DEBUG [M:0;jenkins-hbase4:34965] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 20:08:49,430 INFO [M:0;jenkins-hbase4:34965] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=96.34 KB heapSize=116.35 KB 2023-07-12 20:08:49,444 INFO [M:0;jenkins-hbase4:34965] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=96.34 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/caa094c713794ca48325816b216ae050 2023-07-12 20:08:49,451 DEBUG [M:0;jenkins-hbase4:34965] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/caa094c713794ca48325816b216ae050 as hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/caa094c713794ca48325816b216ae050 2023-07-12 20:08:49,457 INFO [M:0;jenkins-hbase4:34965] regionserver.HStore(1080): Added hdfs://localhost:45739/user/jenkins/test-data/3395219a-f308-556c-2de9-f0466acb23c4/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/caa094c713794ca48325816b216ae050, entries=29, sequenceid=225, filesize=8.3 K 2023-07-12 20:08:49,458 INFO [M:0;jenkins-hbase4:34965] regionserver.HRegion(2948): Finished flush of dataSize ~96.34 KB/98654, heapSize ~116.34 KB/119128, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 28ms, sequenceid=225, compaction requested=false 2023-07-12 20:08:49,459 INFO [M:0;jenkins-hbase4:34965] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 20:08:49,459 DEBUG [M:0;jenkins-hbase4:34965] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 20:08:49,463 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 20:08:49,463 INFO [M:0;jenkins-hbase4:34965] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 20:08:49,463 INFO [M:0;jenkins-hbase4:34965] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34965 2023-07-12 20:08:49,465 DEBUG [M:0;jenkins-hbase4:34965] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,34965,1689192504325 already deleted, retry=false 2023-07-12 20:08:49,686 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 20:08:49,686 INFO [M:0;jenkins-hbase4:34965] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,34965,1689192504325; zookeeper connection closed. 2023-07-12 20:08:49,687 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): master:34965-0x1015b269bb50000, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 20:08:49,787 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): regionserver:37151-0x1015b269bb50008, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 20:08:49,787 INFO [RS:2;jenkins-hbase4:37151] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,37151,1689192510197; zookeeper connection closed. 2023-07-12 20:08:49,787 DEBUG [Listener at localhost/44421-EventThread] zookeeper.ZKWatcher(600): regionserver:37151-0x1015b269bb50008, quorum=127.0.0.1:50668, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 20:08:49,787 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@64f5e6ab] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@64f5e6ab 2023-07-12 20:08:49,787 INFO [Listener at localhost/44421] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-07-12 20:08:49,788 WARN [Listener at localhost/44421] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-12 20:08:49,791 INFO [Listener at localhost/44421] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 20:08:49,896 WARN [BP-1149810363-172.31.14.131-1689192501335 heartbeating to localhost/127.0.0.1:45739] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-12 20:08:49,896 WARN [BP-1149810363-172.31.14.131-1689192501335 heartbeating to localhost/127.0.0.1:45739] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1149810363-172.31.14.131-1689192501335 (Datanode Uuid 266107c9-b366-44ad-b66f-f6bdb7b07e21) service to localhost/127.0.0.1:45739 2023-07-12 20:08:49,898 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/cluster_b448c56c-3a9d-1fce-b944-8d379dcf0ce4/dfs/data/data3/current/BP-1149810363-172.31.14.131-1689192501335] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 20:08:49,898 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/cluster_b448c56c-3a9d-1fce-b944-8d379dcf0ce4/dfs/data/data4/current/BP-1149810363-172.31.14.131-1689192501335] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 20:08:49,900 WARN [Listener at localhost/44421] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-12 20:08:49,902 INFO [Listener at localhost/44421] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 20:08:50,006 WARN [BP-1149810363-172.31.14.131-1689192501335 heartbeating to localhost/127.0.0.1:45739] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-12 20:08:50,006 WARN [BP-1149810363-172.31.14.131-1689192501335 heartbeating to localhost/127.0.0.1:45739] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1149810363-172.31.14.131-1689192501335 (Datanode Uuid 5ec7b338-4711-43ff-baaf-745d20ec4762) service to localhost/127.0.0.1:45739 2023-07-12 20:08:50,007 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/cluster_b448c56c-3a9d-1fce-b944-8d379dcf0ce4/dfs/data/data1/current/BP-1149810363-172.31.14.131-1689192501335] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 20:08:50,007 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3f24302a-7e25-b32d-fcd2-19adfc091cb6/cluster_b448c56c-3a9d-1fce-b944-8d379dcf0ce4/dfs/data/data2/current/BP-1149810363-172.31.14.131-1689192501335] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 20:08:50,047 INFO [Listener at localhost/44421] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 20:08:50,160 INFO [Listener at localhost/44421] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-12 20:08:50,200 INFO [Listener at localhost/44421] hbase.HBaseTestingUtility(1293): Minicluster is down