2023-02-08 01:03:21,898 DEBUG [main] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-02-08 01:03:22,005 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.ipc.TestBlockingIPC timeout: 13 mins 2023-02-08 01:03:22,035 INFO [Time-limited test] hbase.ResourceChecker(147): before: ipc.TestBlockingIPC#testRTEDuringConnectionSetup Thread=8, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=546, ProcessCount=174, AvailableMemoryMB=4232 2023-02-08 01:03:22,168 INFO [Time-limited test] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.SimpleRpcServer hosting hbase.test.pb.TestProtobufRpcProto 2023-02-08 01:03:22,355 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-08 01:03:22,452 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-08 01:03:22,486 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-02-08 01:03:22,592 INFO [Time-limited test] ipc.SimpleRpcServer$Listener(134): testRpcServer: started 3 reader(s) listening on port=44411 2023-02-08 01:03:22,667 DEBUG [Time-limited test] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6e0e876d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-08 01:03:22,675 DEBUG [RpcServer.responder] ipc.SimpleRpcServerResponder(58): RpcServer.responder: starting 2023-02-08 01:03:22,675 INFO [Listener,port=44411] ipc.SimpleRpcServer$Listener(214): Listener,port=44411: starting 2023-02-08 01:03:22,675 INFO [Time-limited test] ipc.FifoRpcScheduler(61): Using FifoRpcScheduler as user call queue; handlerCount=1; maxQueueLength=10 2023-02-08 01:03:22,863 DEBUG [Time-limited test] ipc.RpcConnection(124): Using SIMPLE authentication for service=TestProtobufRpcProto, sasl=false 2023-02-08 01:03:22,929 DEBUG [Time-limited test] ipc.BlockingRpcConnection(453): Connecting to localhost.localdomain/127.0.0.1:44411 2023-02-08 01:03:22,931 DEBUG [Time-limited test] ipc.FailedServers(52): Added failed server with address localhost.localdomain/127.0.0.1:44411 to list caused by java.lang.RuntimeException: Injected fault 2023-02-08 01:03:22,931 DEBUG [Time-limited test] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-08 01:03:22,931 INFO [Time-limited test] ipc.AbstractTestIPC(166): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.io.IOException: Call to address=localhost.localdomain/127.0.0.1:44411 failed on local exception: java.io.IOException: Could not set up IO Streams to localhost.localdomain/127.0.0.1:44411 2023-02-08 01:03:22,932 INFO [Time-limited test] ipc.SimpleRpcServer(438): Stopping server on 44411 2023-02-08 01:03:22,932 INFO [Listener,port=44411] ipc.SimpleRpcServer$Listener(258): Listener,port=44411: stopping 2023-02-08 01:03:22,932 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(158): RpcServer.responder: stopped 2023-02-08 01:03:22,933 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(62): RpcServer.responder: stopping 2023-02-08 01:03:22,942 INFO [Time-limited test] hbase.ResourceChecker(175): after: ipc.TestBlockingIPC#testRTEDuringConnectionSetup Thread=11 (was 8) Potentially hanging thread: Idle-Rpc-Conn-Sweeper-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: HBase-Metrics2-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=261 (was 260) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=546 (was 546), ProcessCount=174 (was 174), AvailableMemoryMB=4091 (was 4232) 2023-02-08 01:03:22,952 INFO [Time-limited test] hbase.ResourceChecker(147): before: ipc.TestBlockingIPC#testTimeout Thread=11, OpenFileDescriptor=261, MaxFileDescriptor=60000, SystemLoadAverage=546, ProcessCount=174, AvailableMemoryMB=4091 2023-02-08 01:03:22,952 INFO [Time-limited test] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.SimpleRpcServer hosting hbase.test.pb.TestProtobufRpcProto 2023-02-08 01:03:22,954 INFO [Time-limited test] ipc.SimpleRpcServer$Listener(134): testRpcServer: started 3 reader(s) listening on port=42113 2023-02-08 01:03:22,954 DEBUG [Time-limited test] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@22360523, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-08 01:03:22,955 DEBUG [RpcServer.responder] ipc.SimpleRpcServerResponder(58): RpcServer.responder: starting 2023-02-08 01:03:22,955 INFO [Time-limited test] ipc.FifoRpcScheduler(61): Using FifoRpcScheduler as user call queue; handlerCount=1; maxQueueLength=10 2023-02-08 01:03:22,955 INFO [Listener,port=42113] ipc.SimpleRpcServer$Listener(214): Listener,port=42113: starting 2023-02-08 01:03:22,958 DEBUG [Time-limited test] ipc.RpcConnection(124): Using SIMPLE authentication for service=TestProtobufRpcProto, sasl=false 2023-02-08 01:03:22,961 DEBUG [Time-limited test] ipc.BlockingRpcConnection(453): Connecting to localhost.localdomain/127.0.0.1:42113 2023-02-08 01:03:22,984 INFO [Reader=1,bindAddress=localhost,port=42113] ipc.ServerRpcConnection(540): Connection from 127.0.0.1:55964, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=TestProtobufRpcProto 2023-02-08 01:03:23,075 INFO [Time-limited test] ipc.AbstractTestIPC(288): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost.localdomain/127.0.0.1:42113 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=0,methodName=pause], waitTime=117ms, rpcTimeout=100ms 2023-02-08 01:03:23,181 INFO [Time-limited test] ipc.AbstractTestIPC(288): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost.localdomain/127.0.0.1:42113 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=1,methodName=pause], waitTime=104ms, rpcTimeout=100ms 2023-02-08 01:03:23,290 INFO [Time-limited test] ipc.AbstractTestIPC(288): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost.localdomain/127.0.0.1:42113 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=2,methodName=pause], waitTime=109ms, rpcTimeout=100ms 2023-02-08 01:03:23,400 INFO [Time-limited test] ipc.AbstractTestIPC(288): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost.localdomain/127.0.0.1:42113 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=3,methodName=pause], waitTime=110ms, rpcTimeout=100ms 2023-02-08 01:03:23,510 INFO [Time-limited test] ipc.AbstractTestIPC(288): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost.localdomain/127.0.0.1:42113 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=4,methodName=pause], waitTime=109ms, rpcTimeout=100ms 2023-02-08 01:03:23,621 INFO [Time-limited test] ipc.AbstractTestIPC(288): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost.localdomain/127.0.0.1:42113 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=5,methodName=pause], waitTime=111ms, rpcTimeout=100ms 2023-02-08 01:03:23,730 INFO [Time-limited test] ipc.AbstractTestIPC(288): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost.localdomain/127.0.0.1:42113 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=6,methodName=pause], waitTime=109ms, rpcTimeout=100ms 2023-02-08 01:03:23,841 INFO [Time-limited test] ipc.AbstractTestIPC(288): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost.localdomain/127.0.0.1:42113 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=7,methodName=pause], waitTime=110ms, rpcTimeout=100ms 2023-02-08 01:03:23,950 INFO [Time-limited test] ipc.AbstractTestIPC(288): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost.localdomain/127.0.0.1:42113 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=8,methodName=pause], waitTime=109ms, rpcTimeout=100ms 2023-02-08 01:03:24,024 WARN [FifoRpcScheduler.handler-pool-0] ipc.CallRunner(105): Dropping timed out call: callId: 1 service: TestProtobufRpcProto methodName: pause size: 20 connection: 127.0.0.1:55964 deadline: 1675818203176 param: TODO: class org.apache.hadoop.hbase.shaded.ipc.protobuf.generated.TestProtos$PauseRequestProto connection: 127.0.0.1:55964 2023-02-08 01:03:24,025 WARN [FifoRpcScheduler.handler-pool-0] ipc.CallRunner(105): Dropping timed out call: callId: 2 service: TestProtobufRpcProto methodName: pause size: 20 connection: 127.0.0.1:55964 deadline: 1675818203281 param: TODO: class org.apache.hadoop.hbase.shaded.ipc.protobuf.generated.TestProtos$PauseRequestProto connection: 127.0.0.1:55964 2023-02-08 01:03:24,025 WARN [FifoRpcScheduler.handler-pool-0] ipc.CallRunner(105): Dropping timed out call: callId: 3 service: TestProtobufRpcProto methodName: pause size: 20 connection: 127.0.0.1:55964 deadline: 1675818203391 param: TODO: class org.apache.hadoop.hbase.shaded.ipc.protobuf.generated.TestProtos$PauseRequestProto connection: 127.0.0.1:55964 2023-02-08 01:03:24,025 WARN [FifoRpcScheduler.handler-pool-0] ipc.CallRunner(105): Dropping timed out call: callId: 4 service: TestProtobufRpcProto methodName: pause size: 20 connection: 127.0.0.1:55964 deadline: 1675818203501 param: TODO: class org.apache.hadoop.hbase.shaded.ipc.protobuf.generated.TestProtos$PauseRequestProto connection: 127.0.0.1:55964 2023-02-08 01:03:24,025 WARN [FifoRpcScheduler.handler-pool-0] ipc.CallRunner(105): Dropping timed out call: callId: 5 service: TestProtobufRpcProto methodName: pause size: 20 connection: 127.0.0.1:55964 deadline: 1675818203611 param: TODO: class org.apache.hadoop.hbase.shaded.ipc.protobuf.generated.TestProtos$PauseRequestProto connection: 127.0.0.1:55964 2023-02-08 01:03:24,025 WARN [FifoRpcScheduler.handler-pool-0] ipc.CallRunner(105): Dropping timed out call: callId: 6 service: TestProtobufRpcProto methodName: pause size: 20 connection: 127.0.0.1:55964 deadline: 1675818203722 param: TODO: class org.apache.hadoop.hbase.shaded.ipc.protobuf.generated.TestProtos$PauseRequestProto connection: 127.0.0.1:55964 2023-02-08 01:03:24,025 WARN [FifoRpcScheduler.handler-pool-0] ipc.CallRunner(105): Dropping timed out call: callId: 7 service: TestProtobufRpcProto methodName: pause size: 20 connection: 127.0.0.1:55964 deadline: 1675818203831 param: TODO: class org.apache.hadoop.hbase.shaded.ipc.protobuf.generated.TestProtos$PauseRequestProto connection: 127.0.0.1:55964 2023-02-08 01:03:24,025 WARN [FifoRpcScheduler.handler-pool-0] ipc.CallRunner(105): Dropping timed out call: callId: 8 service: TestProtobufRpcProto methodName: pause size: 20 connection: 127.0.0.1:55964 deadline: 1675818203942 param: TODO: class org.apache.hadoop.hbase.shaded.ipc.protobuf.generated.TestProtos$PauseRequestProto connection: 127.0.0.1:55964 2023-02-08 01:03:24,061 INFO [Time-limited test] ipc.AbstractTestIPC(288): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost.localdomain/127.0.0.1:42113 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=9,methodName=pause], waitTime=109ms, rpcTimeout=100ms 2023-02-08 01:03:24,061 DEBUG [Time-limited test] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-08 01:03:24,061 INFO [Time-limited test] ipc.SimpleRpcServer(438): Stopping server on 42113 2023-02-08 01:03:24,062 INFO [Listener,port=42113] ipc.SimpleRpcServer$Listener(258): Listener,port=42113: stopping 2023-02-08 01:03:24,062 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(158): RpcServer.responder: stopped 2023-02-08 01:03:24,062 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(62): RpcServer.responder: stopping 2023-02-08 01:03:24,071 INFO [Time-limited test] hbase.ResourceChecker(175): after: ipc.TestBlockingIPC#testTimeout Thread=15 (was 11) Potentially hanging thread: RpcServer.responder java.lang.Throwable.getStackTraceElement(Native Method) java.lang.Throwable.getOurStackTrace(Throwable.java:828) java.lang.Throwable.getStackTrace(Throwable.java:817) org.apache.log4j.spi.LocationInfo.(LocationInfo.java:98) org.apache.log4j.spi.LoggingEvent.getLocationInformation(LoggingEvent.java:260) org.apache.log4j.helpers.PatternParser$ClassNamePatternConverter.getFullyQualifiedName(PatternParser.java:518) org.apache.log4j.helpers.PatternParser$NamedPatternConverter.convert(PatternParser.java:491) org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:63) org.apache.log4j.PatternLayout.format(PatternLayout.java:510) org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:303) org.apache.log4j.WriterAppender.append(WriterAppender.java:156) org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:232) org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:57) org.apache.log4j.Category.callAppenders(Category.java:206) org.apache.log4j.Category.forcedLog(Category.java:384) org.apache.log4j.Category.log(Category.java:810) org.slf4j.impl.Reload4jLoggerAdapter.info(Reload4jLoggerAdapter.java:284) org.apache.hadoop.hbase.ipc.SimpleRpcServerResponder.run(SimpleRpcServerResponder.java:62) Potentially hanging thread: Monitor thread for TaskMonitor java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.monitoring.TaskMonitor$MonitorRunnable.run(TaskMonitor.java:300) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: FifoRpcScheduler.handler-pool-0 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.util.Threads.sleepWithoutInterrupt(Threads.java:152) org.apache.hadoop.hbase.ipc.TestProtobufRpcServiceImpl.pause(TestProtobufRpcServiceImpl.java:110) org.apache.hadoop.hbase.shaded.ipc.protobuf.generated.TestRpcServiceProtos$TestProtobufRpcProto$2.callBlockingMethod(TestRpcServiceProtos.java:144) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:111) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcClient-timer-pool-0 java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:600) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:496) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=264 (was 261) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=546 (was 546), ProcessCount=174 (was 174), AvailableMemoryMB=4084 (was 4091) 2023-02-08 01:03:24,079 INFO [Time-limited test] hbase.ResourceChecker(147): before: ipc.TestBlockingIPC#testRpcScheduler Thread=14, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=546, ProcessCount=174, AvailableMemoryMB=4084 2023-02-08 01:03:24,560 INFO [Time-limited test] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.SimpleRpcServer hosting hbase.test.pb.TestProtobufRpcProto 2023-02-08 01:03:24,567 INFO [Time-limited test] ipc.SimpleRpcServer$Listener(134): testRpcServer: started 3 reader(s) listening on port=45101 2023-02-08 01:03:24,601 DEBUG [Time-limited test] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63818a6a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-08 01:03:24,601 DEBUG [RpcServer.responder] ipc.SimpleRpcServerResponder(58): RpcServer.responder: starting 2023-02-08 01:03:24,601 INFO [Listener,port=45101] ipc.SimpleRpcServer$Listener(214): Listener,port=45101: starting 2023-02-08 01:03:24,602 INFO [Time-limited test] ipc.FifoRpcScheduler(61): Using FifoRpcScheduler$MockitoMock$1234568996 as user call queue; handlerCount=1; maxQueueLength=10 2023-02-08 01:03:24,608 DEBUG [Time-limited test] ipc.RpcConnection(124): Using SIMPLE authentication for service=TestProtobufRpcProto, sasl=false 2023-02-08 01:03:24,609 DEBUG [Time-limited test] ipc.BlockingRpcConnection(453): Connecting to localhost.localdomain/127.0.0.1:45101 2023-02-08 01:03:24,610 INFO [Reader=1,bindAddress=localhost,port=45101] ipc.ServerRpcConnection(540): Connection from 127.0.0.1:45364, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=TestProtobufRpcProto 2023-02-08 01:03:24,623 DEBUG [Time-limited test] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-08 01:03:24,623 INFO [Time-limited test] ipc.SimpleRpcServer(438): Stopping server on 45101 2023-02-08 01:03:24,624 INFO [Listener,port=45101] ipc.SimpleRpcServer$Listener(258): Listener,port=45101: stopping 2023-02-08 01:03:24,624 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(158): RpcServer.responder: stopped 2023-02-08 01:03:24,624 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(62): RpcServer.responder: stopping 2023-02-08 01:03:24,633 INFO [Time-limited test] hbase.ResourceChecker(175): after: ipc.TestBlockingIPC#testRpcScheduler Thread=14 (was 14), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=546 (was 546), ProcessCount=174 (was 174), AvailableMemoryMB=4062 (was 4084) 2023-02-08 01:03:24,639 INFO [Time-limited test] hbase.ResourceChecker(147): before: ipc.TestBlockingIPC#testAsyncRemoteError Thread=14, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=546, ProcessCount=174, AvailableMemoryMB=4062 2023-02-08 01:03:24,639 DEBUG [Time-limited test] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a2d12c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-08 01:03:24,640 INFO [Time-limited test] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.SimpleRpcServer hosting hbase.test.pb.TestProtobufRpcProto 2023-02-08 01:03:24,641 INFO [Time-limited test] ipc.SimpleRpcServer$Listener(134): testRpcServer: started 3 reader(s) listening on port=42997 2023-02-08 01:03:24,641 DEBUG [RpcServer.responder] ipc.SimpleRpcServerResponder(58): RpcServer.responder: starting 2023-02-08 01:03:24,641 INFO [Listener,port=42997] ipc.SimpleRpcServer$Listener(214): Listener,port=42997: starting 2023-02-08 01:03:24,641 INFO [Time-limited test] ipc.FifoRpcScheduler(61): Using FifoRpcScheduler as user call queue; handlerCount=1; maxQueueLength=10 2023-02-08 01:03:24,643 DEBUG [Time-limited test] ipc.RpcConnection(124): Using SIMPLE authentication for service=TestProtobufRpcProto, sasl=false 2023-02-08 01:03:24,643 DEBUG [Time-limited test] ipc.BlockingRpcConnection(453): Connecting to localhost.localdomain/127.0.0.1:42997 2023-02-08 01:03:24,644 INFO [Reader=1,bindAddress=localhost,port=42997] ipc.ServerRpcConnection(540): Connection from 127.0.0.1:50900, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=TestProtobufRpcProto 2023-02-08 01:03:24,648 DEBUG [FifoRpcScheduler.handler-pool-0] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.DoNotRetryIOException: server error! at org.apache.hadoop.hbase.ipc.TestProtobufRpcServiceImpl.error(TestProtobufRpcServiceImpl.java:104) at org.apache.hadoop.hbase.shaded.ipc.protobuf.generated.TestRpcServiceProtos$TestProtobufRpcProto$2.callBlockingMethod(TestRpcServiceProtos.java:142) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:111) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-02-08 01:03:24,650 DEBUG [FifoRpcScheduler.handler-pool-0] ipc.CallRunner(144): callId: 0 service: TestProtobufRpcProto methodName: error size: 17 connection: 127.0.0.1:50900 deadline: 9223372036854775807, exception=org.apache.hadoop.hbase.DoNotRetryIOException: server error! 2023-02-08 01:03:24,652 INFO [Time-limited test] ipc.AbstractTestIPC(369): Caught expected exception: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.DoNotRetryIOException): org.apache.hadoop.hbase.DoNotRetryIOException: server error! at org.apache.hadoop.hbase.ipc.TestProtobufRpcServiceImpl.error(TestProtobufRpcServiceImpl.java:104) at org.apache.hadoop.hbase.shaded.ipc.protobuf.generated.TestRpcServiceProtos$TestProtobufRpcProto$2.callBlockingMethod(TestRpcServiceProtos.java:142) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:111) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-02-08 01:03:24,883 DEBUG [Time-limited test] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-08 01:03:24,883 INFO [Time-limited test] ipc.SimpleRpcServer(438): Stopping server on 42997 2023-02-08 01:03:24,883 INFO [Listener,port=42997] ipc.SimpleRpcServer$Listener(258): Listener,port=42997: stopping 2023-02-08 01:03:24,883 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(158): RpcServer.responder: stopped 2023-02-08 01:03:24,884 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(62): RpcServer.responder: stopping 2023-02-08 01:03:24,890 INFO [Time-limited test] hbase.ResourceChecker(175): after: ipc.TestBlockingIPC#testAsyncRemoteError Thread=17 (was 14) Potentially hanging thread: RpcServer idle connection scanner for port 42997 java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3693) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.responder java.lang.Thread.isInterrupted(Native Method) java.lang.Thread.interrupted(Thread.java:952) org.apache.maven.surefire.booter.spi.EventChannelEncoder.write(EventChannelEncoder.java:304) org.apache.maven.surefire.booter.spi.EventChannelEncoder.setOutErr(EventChannelEncoder.java:189) org.apache.maven.surefire.booter.spi.EventChannelEncoder.testOutput(EventChannelEncoder.java:183) org.apache.maven.surefire.api.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.java:113) org.apache.maven.surefire.api.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.java:44) org.apache.maven.surefire.common.junit4.JUnit4RunListener.writeTestOutput(JUnit4RunListener.java:235) org.apache.maven.surefire.api.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleOutputCapture.java:64) sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291) sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295) sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141) java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229) org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:54) org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:317) org.apache.log4j.WriterAppender.append(WriterAppender.java:156) org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:232) org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:57) org.apache.log4j.Category.callAppenders(Category.java:206) org.apache.log4j.Category.forcedLog(Category.java:384) org.apache.log4j.Category.log(Category.java:810) org.slf4j.impl.Reload4jLoggerAdapter.info(Reload4jLoggerAdapter.java:284) org.apache.hadoop.hbase.ipc.SimpleRpcServerResponder.doRunLoop(SimpleRpcServerResponder.java:158) org.apache.hadoop.hbase.ipc.SimpleRpcServerResponder.run(SimpleRpcServerResponder.java:60) - Thread LEAK? -, OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=546 (was 546), ProcessCount=174 (was 174), AvailableMemoryMB=4037 (was 4062) 2023-02-08 01:03:24,897 INFO [Time-limited test] hbase.ResourceChecker(147): before: ipc.TestBlockingIPC#testAsyncEcho Thread=15, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=546, ProcessCount=174, AvailableMemoryMB=4036 2023-02-08 01:03:24,919 INFO [Time-limited test] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.SimpleRpcServer hosting hbase.test.pb.TestProtobufRpcProto 2023-02-08 01:03:24,920 INFO [Time-limited test] ipc.SimpleRpcServer$Listener(134): testRpcServer: started 3 reader(s) listening on port=42419 2023-02-08 01:03:24,921 DEBUG [Time-limited test] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49c1e570, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-08 01:03:24,921 DEBUG [RpcServer.responder] ipc.SimpleRpcServerResponder(58): RpcServer.responder: starting 2023-02-08 01:03:24,921 INFO [Listener,port=42419] ipc.SimpleRpcServer$Listener(214): Listener,port=42419: starting 2023-02-08 01:03:24,921 INFO [Time-limited test] ipc.FifoRpcScheduler(61): Using FifoRpcScheduler as user call queue; handlerCount=1; maxQueueLength=10 2023-02-08 01:03:24,922 DEBUG [Time-limited test] ipc.RpcConnection(124): Using SIMPLE authentication for service=TestProtobufRpcProto, sasl=false 2023-02-08 01:03:24,922 DEBUG [Time-limited test] ipc.BlockingRpcConnection(453): Connecting to localhost.localdomain/127.0.0.1:42419 2023-02-08 01:03:24,923 INFO [Reader=1,bindAddress=localhost,port=42419] ipc.ServerRpcConnection(540): Connection from 127.0.0.1:60822, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=TestProtobufRpcProto 2023-02-08 01:03:24,927 DEBUG [Time-limited test] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-08 01:03:24,927 INFO [Time-limited test] ipc.SimpleRpcServer(438): Stopping server on 42419 2023-02-08 01:03:24,927 INFO [Listener,port=42419] ipc.SimpleRpcServer$Listener(258): Listener,port=42419: stopping 2023-02-08 01:03:24,928 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(158): RpcServer.responder: stopped 2023-02-08 01:03:24,928 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(62): RpcServer.responder: stopping 2023-02-08 01:03:24,937 INFO [Time-limited test] hbase.ResourceChecker(175): after: ipc.TestBlockingIPC#testAsyncEcho Thread=16 (was 15) Potentially hanging thread: RpcServer.responder java.lang.Double.doubleToRawLongBits(Native Method) java.lang.Math.getExponent(Math.java:1840) java.lang.StrictMath.floorOrCeil(StrictMath.java:374) java.lang.StrictMath.ceil(StrictMath.java:340) java.lang.Math.ceil(Math.java:423) org.apache.maven.surefire.api.stream.AbstractStreamEncoder.estimateBufferLength(AbstractStreamEncoder.java:206) org.apache.maven.surefire.booter.spi.EventChannelEncoder.encodeMessage(EventChannelEncoder.java:395) org.apache.maven.surefire.booter.spi.EventChannelEncoder.setOutErr(EventChannelEncoder.java:188) org.apache.maven.surefire.booter.spi.EventChannelEncoder.testOutput(EventChannelEncoder.java:183) org.apache.maven.surefire.api.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.java:113) org.apache.maven.surefire.api.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.java:44) org.apache.maven.surefire.common.junit4.JUnit4RunListener.writeTestOutput(JUnit4RunListener.java:235) org.apache.maven.surefire.api.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleOutputCapture.java:64) sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291) sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295) sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141) java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229) org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:54) org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:317) org.apache.log4j.WriterAppender.append(WriterAppender.java:156) org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:232) org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:57) org.apache.log4j.Category.callAppenders(Category.java:206) org.apache.log4j.Category.forcedLog(Category.java:384) org.apache.log4j.Category.log(Category.java:810) org.slf4j.impl.Reload4jLoggerAdapter.info(Reload4jLoggerAdapter.java:284) org.apache.hadoop.hbase.ipc.SimpleRpcServerResponder.run(SimpleRpcServerResponder.java:62) - Thread LEAK? -, OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=546 (was 546), ProcessCount=174 (was 174), AvailableMemoryMB=4034 (was 4036) 2023-02-08 01:03:24,944 INFO [Time-limited test] hbase.ResourceChecker(147): before: ipc.TestBlockingIPC#testConnectionCloseWithOutstandingRPCs Thread=15, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=546, ProcessCount=174, AvailableMemoryMB=4033 2023-02-08 01:03:24,947 INFO [Time-limited test] ipc.SimpleRpcServer$Listener(134): testRpcServer: started 3 reader(s) listening on port=43473 2023-02-08 01:03:24,948 DEBUG [Time-limited test] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@61356683, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-08 01:03:24,948 DEBUG [RpcServer.responder] ipc.SimpleRpcServerResponder(58): RpcServer.responder: starting 2023-02-08 01:03:24,948 INFO [Listener,port=43473] ipc.SimpleRpcServer$Listener(214): Listener,port=43473: starting 2023-02-08 01:03:24,948 INFO [Time-limited test] ipc.FifoRpcScheduler(61): Using FifoRpcScheduler as user call queue; handlerCount=1; maxQueueLength=10 2023-02-08 01:03:24,949 DEBUG [Time-limited test] ipc.RpcConnection(124): Using SIMPLE authentication for service=TestProtobufRpcProto, sasl=false 2023-02-08 01:03:24,949 DEBUG [Time-limited test] ipc.BlockingRpcConnection(453): Connecting to localhost.localdomain/127.0.0.1:43473 2023-02-08 01:03:24,950 INFO [Reader=1,bindAddress=localhost,port=43473] ipc.ServerRpcConnection(540): Connection from 127.0.0.1:48744, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=TestProtobufRpcProto 2023-02-08 01:03:24,951 DEBUG [Reader=1,bindAddress=localhost,port=43473] ipc.SimpleRpcServer$Listener(326): Caught exception while reading: org.apache.hadoop.hbase.DoNotRetryIOException: Failing for test at org.apache.hadoop.hbase.ipc.TestBlockingIPC$TestFailingRpcServer$FailingConnection.processRequest(TestBlockingIPC.java:94) at org.apache.hadoop.hbase.ipc.ServerRpcConnection.processOneRpc(ServerRpcConnection.java:444) at org.apache.hadoop.hbase.ipc.SimpleServerRpcConnection.process(SimpleServerRpcConnection.java:286) at org.apache.hadoop.hbase.ipc.SimpleServerRpcConnection.readAndProcess(SimpleServerRpcConnection.java:252) at org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener.doRead(SimpleRpcServer.java:319) at org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener$Reader.doRunLoop(SimpleRpcServer.java:181) at org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener$Reader.run(SimpleRpcServer.java:154) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-02-08 01:03:24,951 DEBUG [Time-limited test] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-08 01:03:24,952 INFO [Time-limited test] ipc.AbstractTestIPC(319): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.io.EOFException: Call to address=localhost.localdomain/127.0.0.1:43473 failed on local exception: java.io.EOFException 2023-02-08 01:03:24,952 INFO [Time-limited test] ipc.SimpleRpcServer(438): Stopping server on 43473 2023-02-08 01:03:24,952 INFO [Listener,port=43473] ipc.SimpleRpcServer$Listener(258): Listener,port=43473: stopping 2023-02-08 01:03:24,952 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(158): RpcServer.responder: stopped 2023-02-08 01:03:24,952 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(62): RpcServer.responder: stopping 2023-02-08 01:03:24,961 INFO [Time-limited test] hbase.ResourceChecker(175): after: ipc.TestBlockingIPC#testConnectionCloseWithOutstandingRPCs Thread=16 (was 15) Potentially hanging thread: RpcServer.responder java.lang.Throwable.getStackTraceElement(Native Method) java.lang.Throwable.getOurStackTrace(Throwable.java:828) java.lang.Throwable.getStackTrace(Throwable.java:817) org.apache.log4j.spi.LocationInfo.(LocationInfo.java:98) org.apache.log4j.spi.LoggingEvent.getLocationInformation(LoggingEvent.java:260) org.apache.log4j.helpers.PatternParser$ClassNamePatternConverter.getFullyQualifiedName(PatternParser.java:518) org.apache.log4j.helpers.PatternParser$NamedPatternConverter.convert(PatternParser.java:491) org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:63) org.apache.log4j.PatternLayout.format(PatternLayout.java:510) org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:303) org.apache.log4j.WriterAppender.append(WriterAppender.java:156) org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:232) org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:57) org.apache.log4j.Category.callAppenders(Category.java:206) org.apache.log4j.Category.forcedLog(Category.java:384) org.apache.log4j.Category.log(Category.java:810) org.slf4j.impl.Reload4jLoggerAdapter.info(Reload4jLoggerAdapter.java:284) org.apache.hadoop.hbase.ipc.SimpleRpcServerResponder.run(SimpleRpcServerResponder.java:62) - Thread LEAK? -, OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=546 (was 546), ProcessCount=174 (was 174), AvailableMemoryMB=4035 (was 4033) - AvailableMemoryMB LEAK? - 2023-02-08 01:03:24,969 INFO [Time-limited test] hbase.ResourceChecker(147): before: ipc.TestBlockingIPC#testCompressCellBlock Thread=15, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=546, ProcessCount=174, AvailableMemoryMB=4035 2023-02-08 01:03:24,982 INFO [Time-limited test] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.SimpleRpcServer hosting hbase.test.pb.TestProtobufRpcProto 2023-02-08 01:03:24,985 INFO [Time-limited test] ipc.SimpleRpcServer$Listener(134): testRpcServer: started 3 reader(s) listening on port=38013 2023-02-08 01:03:24,988 DEBUG [Time-limited test] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63f52716, compressor=org.apache.hadoop.io.compress.GzipCodec@754e0973, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-08 01:03:24,988 INFO [Time-limited test] ipc.FifoRpcScheduler(61): Using FifoRpcScheduler as user call queue; handlerCount=1; maxQueueLength=10 2023-02-08 01:03:24,988 INFO [Listener,port=38013] ipc.SimpleRpcServer$Listener(214): Listener,port=38013: starting 2023-02-08 01:03:24,988 DEBUG [RpcServer.responder] ipc.SimpleRpcServerResponder(58): RpcServer.responder: starting 2023-02-08 01:03:24,989 DEBUG [Time-limited test] ipc.RpcConnection(124): Using SIMPLE authentication for service=TestProtobufRpcProto, sasl=false 2023-02-08 01:03:25,013 DEBUG [Time-limited test] ipc.BlockingRpcConnection(453): Connecting to localhost.localdomain/127.0.0.1:38013 2023-02-08 01:03:25,014 INFO [Reader=1,bindAddress=localhost,port=38013] ipc.ServerRpcConnection(540): Connection from 127.0.0.1:59996, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=TestProtobufRpcProto 2023-02-08 01:03:25,026 DEBUG [FifoRpcScheduler.handler-pool-0] ipc.SimpleRpcServerResponder(238): 127.0.0.1:55964: output error -- closing 2023-02-08 01:03:25,026 WARN [FifoRpcScheduler.handler-pool-0] ipc.CallRunner(181): FifoRpcScheduler.handler-pool-0: caught a ClosedChannelException, this means that the server localhost.localdomain/127.0.0.1:42113 was processing a request but the client went away. The error message was: null 2023-02-08 01:03:25,038 DEBUG [Time-limited test] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-08 01:03:25,038 INFO [Time-limited test] ipc.SimpleRpcServer(438): Stopping server on 38013 2023-02-08 01:03:25,038 INFO [Listener,port=38013] ipc.SimpleRpcServer$Listener(258): Listener,port=38013: stopping 2023-02-08 01:03:25,038 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(158): RpcServer.responder: stopped 2023-02-08 01:03:25,039 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(62): RpcServer.responder: stopping 2023-02-08 01:03:25,052 INFO [Time-limited test] hbase.ResourceChecker(175): after: ipc.TestBlockingIPC#testCompressCellBlock Thread=21 (was 15) Potentially hanging thread: Listener,port=38013 org.apache.log4j.Category.callAppenders(Category.java:211) org.apache.log4j.Category.forcedLog(Category.java:384) org.apache.log4j.Category.log(Category.java:810) org.slf4j.impl.Reload4jLoggerAdapter.info(Reload4jLoggerAdapter.java:284) org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener.run(SimpleRpcServer.java:258) Potentially hanging thread: Reader=2,bindAddress=localhost,port=38013 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener$Reader.doRunLoop(SimpleRpcServer.java:174) org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener$Reader.run(SimpleRpcServer.java:154) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Reader=0,bindAddress=localhost,port=38013 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener$Reader.doRunLoop(SimpleRpcServer.java:174) org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener$Reader.run(SimpleRpcServer.java:154) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer idle connection scanner for port 38013 java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: BRPC Connection (94012143) to localhost.localdomain/127.0.0.1:38013 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.ipc.BlockingRpcConnection.waitForWork(BlockingRpcConnection.java:337) org.apache.hadoop.hbase.ipc.BlockingRpcConnection.run(BlockingRpcConnection.java:348) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.responder - Thread LEAK? -, OpenFileDescriptor=267 (was 264) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=546 (was 546), ProcessCount=174 (was 174), AvailableMemoryMB=4030 (was 4035) 2023-02-08 01:03:25,059 INFO [Time-limited test] hbase.ResourceChecker(147): before: ipc.TestBlockingIPC#testRpcServerForNotNullRemoteAddressInCallObject Thread=14, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=546, ProcessCount=174, AvailableMemoryMB=4030 2023-02-08 01:03:25,059 INFO [Time-limited test] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.SimpleRpcServer hosting hbase.test.pb.TestProtobufRpcProto 2023-02-08 01:03:25,060 INFO [Time-limited test] ipc.SimpleRpcServer$Listener(134): testRpcServer: started 3 reader(s) listening on port=41319 2023-02-08 01:03:25,061 DEBUG [Time-limited test] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44e94efd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-08 01:03:25,061 DEBUG [RpcServer.responder] ipc.SimpleRpcServerResponder(58): RpcServer.responder: starting 2023-02-08 01:03:25,061 INFO [Listener,port=41319] ipc.SimpleRpcServer$Listener(214): Listener,port=41319: starting 2023-02-08 01:03:25,061 INFO [Time-limited test] ipc.FifoRpcScheduler(61): Using FifoRpcScheduler as user call queue; handlerCount=1; maxQueueLength=10 2023-02-08 01:03:25,064 DEBUG [Time-limited test] ipc.RpcConnection(124): Using SIMPLE authentication for service=TestProtobufRpcProto, sasl=false 2023-02-08 01:03:25,064 DEBUG [Time-limited test] ipc.BlockingRpcConnection(453): Connecting to localhost.localdomain/127.0.0.1:41319 2023-02-08 01:03:25,065 INFO [Reader=1,bindAddress=localhost,port=41319] ipc.ServerRpcConnection(540): Connection from 127.0.0.1:47180, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=TestProtobufRpcProto 2023-02-08 01:03:25,068 DEBUG [Time-limited test] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-08 01:03:25,068 INFO [Time-limited test] ipc.SimpleRpcServer(438): Stopping server on 41319 2023-02-08 01:03:25,068 INFO [Listener,port=41319] ipc.SimpleRpcServer$Listener(258): Listener,port=41319: stopping 2023-02-08 01:03:25,069 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(158): RpcServer.responder: stopped 2023-02-08 01:03:25,069 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(62): RpcServer.responder: stopping 2023-02-08 01:03:25,078 INFO [Time-limited test] hbase.ResourceChecker(175): after: ipc.TestBlockingIPC#testRpcServerForNotNullRemoteAddressInCallObject Thread=15 (was 14) - Thread LEAK? -, OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=546 (was 546), ProcessCount=174 (was 174), AvailableMemoryMB=4029 (was 4030) 2023-02-08 01:03:25,087 INFO [Time-limited test] hbase.ResourceChecker(147): before: ipc.TestBlockingIPC#testRemoteError Thread=14, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=546, ProcessCount=174, AvailableMemoryMB=4029 2023-02-08 01:03:25,087 INFO [Time-limited test] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.SimpleRpcServer hosting hbase.test.pb.TestProtobufRpcProto 2023-02-08 01:03:25,088 INFO [Time-limited test] ipc.SimpleRpcServer$Listener(134): testRpcServer: started 3 reader(s) listening on port=32857 2023-02-08 01:03:25,089 DEBUG [Time-limited test] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6639087e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-08 01:03:25,089 DEBUG [RpcServer.responder] ipc.SimpleRpcServerResponder(58): RpcServer.responder: starting 2023-02-08 01:03:25,091 INFO [Time-limited test] ipc.FifoRpcScheduler(61): Using FifoRpcScheduler as user call queue; handlerCount=1; maxQueueLength=10 2023-02-08 01:03:25,091 INFO [Listener,port=32857] ipc.SimpleRpcServer$Listener(214): Listener,port=32857: starting 2023-02-08 01:03:25,091 DEBUG [Time-limited test] ipc.RpcConnection(124): Using SIMPLE authentication for service=TestProtobufRpcProto, sasl=false 2023-02-08 01:03:25,091 DEBUG [Time-limited test] ipc.BlockingRpcConnection(453): Connecting to localhost.localdomain/127.0.0.1:32857 2023-02-08 01:03:25,092 INFO [Reader=1,bindAddress=localhost,port=32857] ipc.ServerRpcConnection(540): Connection from 127.0.0.1:56374, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=TestProtobufRpcProto 2023-02-08 01:03:25,093 DEBUG [FifoRpcScheduler.handler-pool-0] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.DoNotRetryIOException: server error! at org.apache.hadoop.hbase.ipc.TestProtobufRpcServiceImpl.error(TestProtobufRpcServiceImpl.java:104) at org.apache.hadoop.hbase.shaded.ipc.protobuf.generated.TestRpcServiceProtos$TestProtobufRpcProto$2.callBlockingMethod(TestRpcServiceProtos.java:142) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:111) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-02-08 01:03:25,093 DEBUG [FifoRpcScheduler.handler-pool-0] ipc.CallRunner(144): callId: 0 service: TestProtobufRpcProto methodName: error size: 17 connection: 127.0.0.1:56374 deadline: 9223372036854775807, exception=org.apache.hadoop.hbase.DoNotRetryIOException: server error! 2023-02-08 01:03:25,094 DEBUG [Time-limited test] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-08 01:03:25,094 INFO [Time-limited test] ipc.AbstractTestIPC(259): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.DoNotRetryIOException): org.apache.hadoop.hbase.DoNotRetryIOException: server error! at org.apache.hadoop.hbase.ipc.TestProtobufRpcServiceImpl.error(TestProtobufRpcServiceImpl.java:104) at org.apache.hadoop.hbase.shaded.ipc.protobuf.generated.TestRpcServiceProtos$TestProtobufRpcProto$2.callBlockingMethod(TestRpcServiceProtos.java:142) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:111) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-02-08 01:03:25,094 INFO [Time-limited test] ipc.SimpleRpcServer(438): Stopping server on 32857 2023-02-08 01:03:25,094 INFO [Listener,port=32857] ipc.SimpleRpcServer$Listener(258): Listener,port=32857: stopping 2023-02-08 01:03:25,094 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(158): RpcServer.responder: stopped 2023-02-08 01:03:25,095 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(62): RpcServer.responder: stopping 2023-02-08 01:03:25,101 INFO [Time-limited test] hbase.ResourceChecker(175): after: ipc.TestBlockingIPC#testRemoteError Thread=17 (was 14) Potentially hanging thread: RpcServer idle connection scanner for port 32857 java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: Listener,port=32857 java.lang.Thread.isInterrupted(Native Method) java.lang.Thread.interrupted(Thread.java:952) org.apache.maven.surefire.booter.spi.EventChannelEncoder.write(EventChannelEncoder.java:304) org.apache.maven.surefire.booter.spi.EventChannelEncoder.setOutErr(EventChannelEncoder.java:189) org.apache.maven.surefire.booter.spi.EventChannelEncoder.testOutput(EventChannelEncoder.java:183) org.apache.maven.surefire.api.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.java:113) org.apache.maven.surefire.api.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.java:44) org.apache.maven.surefire.common.junit4.JUnit4RunListener.writeTestOutput(JUnit4RunListener.java:235) org.apache.maven.surefire.api.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleOutputCapture.java:64) sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291) sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295) sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141) java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229) org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:54) org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:317) org.apache.log4j.WriterAppender.append(WriterAppender.java:156) org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:232) org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:57) org.apache.log4j.Category.callAppenders(Category.java:206) org.apache.log4j.Category.forcedLog(Category.java:384) org.apache.log4j.Category.log(Category.java:810) org.slf4j.impl.Reload4jLoggerAdapter.info(Reload4jLoggerAdapter.java:284) org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener.run(SimpleRpcServer.java:258) Potentially hanging thread: RpcServer.responder org.apache.log4j.Category.callAppenders(Category.java:205) org.apache.log4j.Category.forcedLog(Category.java:384) org.apache.log4j.Category.log(Category.java:810) org.slf4j.impl.Reload4jLoggerAdapter.info(Reload4jLoggerAdapter.java:284) org.apache.hadoop.hbase.ipc.SimpleRpcServerResponder.doRunLoop(SimpleRpcServerResponder.java:158) org.apache.hadoop.hbase.ipc.SimpleRpcServerResponder.run(SimpleRpcServerResponder.java:60) - Thread LEAK? -, OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=546 (was 546), ProcessCount=174 (was 174), AvailableMemoryMB=4029 (was 4029) 2023-02-08 01:03:25,107 INFO [Time-limited test] hbase.ResourceChecker(147): before: ipc.TestBlockingIPC#testNoCodec Thread=14, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=546, ProcessCount=174, AvailableMemoryMB=4028 2023-02-08 01:03:25,117 INFO [Time-limited test] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.SimpleRpcServer hosting hbase.test.pb.TestProtobufRpcProto 2023-02-08 01:03:25,118 INFO [Time-limited test] ipc.SimpleRpcServer$Listener(134): testRpcServer: started 3 reader(s) listening on port=33563 2023-02-08 01:03:25,119 DEBUG [Time-limited test] ipc.AbstractRpcClient(190): Codec=null, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-08 01:03:25,119 DEBUG [RpcServer.responder] ipc.SimpleRpcServerResponder(58): RpcServer.responder: starting 2023-02-08 01:03:25,119 INFO [Time-limited test] ipc.FifoRpcScheduler(61): Using FifoRpcScheduler as user call queue; handlerCount=1; maxQueueLength=10 2023-02-08 01:03:25,119 INFO [Listener,port=33563] ipc.SimpleRpcServer$Listener(214): Listener,port=33563: starting 2023-02-08 01:03:25,120 DEBUG [Time-limited test] ipc.RpcConnection(124): Using SIMPLE authentication for service=TestProtobufRpcProto, sasl=false 2023-02-08 01:03:25,120 DEBUG [Time-limited test] ipc.BlockingRpcConnection(453): Connecting to localhost.localdomain/127.0.0.1:33563 2023-02-08 01:03:25,120 INFO [Reader=1,bindAddress=localhost,port=33563] ipc.ServerRpcConnection(540): Connection from 127.0.0.1:45662, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=TestProtobufRpcProto 2023-02-08 01:03:25,124 DEBUG [Time-limited test] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-08 01:03:25,124 INFO [Time-limited test] ipc.SimpleRpcServer(438): Stopping server on 33563 2023-02-08 01:03:25,124 INFO [Listener,port=33563] ipc.SimpleRpcServer$Listener(258): Listener,port=33563: stopping 2023-02-08 01:03:25,124 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(158): RpcServer.responder: stopped 2023-02-08 01:03:25,124 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(62): RpcServer.responder: stopping 2023-02-08 01:03:25,131 INFO [Time-limited test] hbase.ResourceChecker(175): after: ipc.TestBlockingIPC#testNoCodec Thread=15 (was 14) - Thread LEAK? -, OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=546 (was 546), ProcessCount=174 (was 174), AvailableMemoryMB=4027 (was 4028) 2023-02-08 01:03:25,137 INFO [Time-limited test] hbase.ResourceChecker(147): before: ipc.TestBlockingIPC#testRpcMaxRequestSize Thread=14, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=546, ProcessCount=174, AvailableMemoryMB=4027 2023-02-08 01:03:25,138 INFO [Time-limited test] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.SimpleRpcServer hosting hbase.test.pb.TestProtobufRpcProto 2023-02-08 01:03:25,139 INFO [Time-limited test] ipc.SimpleRpcServer$Listener(134): testRpcServer: started 3 reader(s) listening on port=45127 2023-02-08 01:03:25,139 DEBUG [Time-limited test] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54566c03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-08 01:03:25,139 DEBUG [RpcServer.responder] ipc.SimpleRpcServerResponder(58): RpcServer.responder: starting 2023-02-08 01:03:25,139 INFO [Listener,port=45127] ipc.SimpleRpcServer$Listener(214): Listener,port=45127: starting 2023-02-08 01:03:25,139 INFO [Time-limited test] ipc.FifoRpcScheduler(61): Using FifoRpcScheduler as user call queue; handlerCount=1; maxQueueLength=10 2023-02-08 01:03:25,147 DEBUG [Time-limited test] ipc.RpcConnection(124): Using SIMPLE authentication for service=TestProtobufRpcProto, sasl=false 2023-02-08 01:03:25,147 DEBUG [Time-limited test] ipc.BlockingRpcConnection(453): Connecting to localhost.localdomain/127.0.0.1:45127 2023-02-08 01:03:25,148 INFO [Reader=1,bindAddress=localhost,port=45127] ipc.ServerRpcConnection(540): Connection from 127.0.0.1:50748, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=TestProtobufRpcProto 2023-02-08 01:03:25,148 WARN [Reader=1,bindAddress=localhost,port=45127] ipc.SimpleServerRpcConnection(187): RPC data length of 1260 received from 127.0.0.1 is greater than max allowed 1000. Set "hbase.ipc.max.request.size" on server to override this limit (not recommended) 2023-02-08 01:03:25,154 DEBUG [Time-limited test] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-08 01:03:25,154 INFO [Time-limited test] ipc.AbstractTestIPC(220): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.RequestTooBigException): RPC data length of 1260 received from 127.0.0.1 is greater than max allowed 1000. Set "hbase.ipc.max.request.size" on server to override this limit (not recommended) 2023-02-08 01:03:25,154 INFO [Time-limited test] ipc.SimpleRpcServer(438): Stopping server on 45127 2023-02-08 01:03:25,154 INFO [Listener,port=45127] ipc.SimpleRpcServer$Listener(258): Listener,port=45127: stopping 2023-02-08 01:03:25,154 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(158): RpcServer.responder: stopped 2023-02-08 01:03:25,155 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(62): RpcServer.responder: stopping 2023-02-08 01:03:25,161 INFO [Time-limited test] hbase.ResourceChecker(175): after: ipc.TestBlockingIPC#testRpcMaxRequestSize Thread=15 (was 14) - Thread LEAK? -, OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=546 (was 546), ProcessCount=174 (was 174), AvailableMemoryMB=4027 (was 4027) 2023-02-08 01:03:25,169 INFO [Time-limited test] hbase.ResourceChecker(147): before: ipc.TestBlockingIPC#testAsyncTimeout Thread=14, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=546, ProcessCount=174, AvailableMemoryMB=4026 2023-02-08 01:03:25,169 INFO [Time-limited test] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.SimpleRpcServer hosting hbase.test.pb.TestProtobufRpcProto 2023-02-08 01:03:25,170 INFO [Time-limited test] ipc.SimpleRpcServer$Listener(134): testRpcServer: started 3 reader(s) listening on port=46245 2023-02-08 01:03:25,171 DEBUG [Time-limited test] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5524fdaa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-08 01:03:25,171 DEBUG [RpcServer.responder] ipc.SimpleRpcServerResponder(58): RpcServer.responder: starting 2023-02-08 01:03:25,171 INFO [Listener,port=46245] ipc.SimpleRpcServer$Listener(214): Listener,port=46245: starting 2023-02-08 01:03:25,171 INFO [Time-limited test] ipc.FifoRpcScheduler(61): Using FifoRpcScheduler as user call queue; handlerCount=1; maxQueueLength=10 2023-02-08 01:03:25,172 DEBUG [Time-limited test] ipc.RpcConnection(124): Using SIMPLE authentication for service=TestProtobufRpcProto, sasl=false 2023-02-08 01:03:25,172 DEBUG [Time-limited test] ipc.BlockingRpcConnection(453): Connecting to localhost.localdomain/127.0.0.1:46245 2023-02-08 01:03:25,173 INFO [Reader=1,bindAddress=localhost,port=46245] ipc.ServerRpcConnection(540): Connection from 127.0.0.1:51352, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=TestProtobufRpcProto 2023-02-08 01:03:25,281 INFO [Time-limited test] ipc.AbstractTestIPC(406): Caught expected exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost.localdomain/127.0.0.1:46245 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=0,methodName=pause], waitTime=109ms, rpcTimeout=100ms 2023-02-08 01:03:25,281 INFO [Time-limited test] ipc.AbstractTestIPC(406): Caught expected exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost.localdomain/127.0.0.1:46245 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=1,methodName=pause], waitTime=107ms, rpcTimeout=100ms 2023-02-08 01:03:25,281 INFO [Time-limited test] ipc.AbstractTestIPC(406): Caught expected exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost.localdomain/127.0.0.1:46245 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=2,methodName=pause], waitTime=107ms, rpcTimeout=100ms 2023-02-08 01:03:25,281 INFO [Time-limited test] ipc.AbstractTestIPC(406): Caught expected exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost.localdomain/127.0.0.1:46245 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=3,methodName=pause], waitTime=107ms, rpcTimeout=100ms 2023-02-08 01:03:25,281 INFO [Time-limited test] ipc.AbstractTestIPC(406): Caught expected exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost.localdomain/127.0.0.1:46245 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=4,methodName=pause], waitTime=107ms, rpcTimeout=100ms 2023-02-08 01:03:25,281 INFO [Time-limited test] ipc.AbstractTestIPC(406): Caught expected exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost.localdomain/127.0.0.1:46245 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=5,methodName=pause], waitTime=108ms, rpcTimeout=100ms 2023-02-08 01:03:25,282 INFO [Time-limited test] ipc.AbstractTestIPC(406): Caught expected exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost.localdomain/127.0.0.1:46245 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=6,methodName=pause], waitTime=108ms, rpcTimeout=100ms 2023-02-08 01:03:25,282 INFO [Time-limited test] ipc.AbstractTestIPC(406): Caught expected exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost.localdomain/127.0.0.1:46245 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=7,methodName=pause], waitTime=108ms, rpcTimeout=100ms 2023-02-08 01:03:25,282 INFO [Time-limited test] ipc.AbstractTestIPC(406): Caught expected exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost.localdomain/127.0.0.1:46245 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=8,methodName=pause], waitTime=108ms, rpcTimeout=100ms 2023-02-08 01:03:25,282 INFO [Time-limited test] ipc.AbstractTestIPC(406): Caught expected exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost.localdomain/127.0.0.1:46245 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=9,methodName=pause], waitTime=107ms, rpcTimeout=100ms 2023-02-08 01:03:25,282 DEBUG [Time-limited test] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-08 01:03:25,282 INFO [Time-limited test] ipc.SimpleRpcServer(438): Stopping server on 46245 2023-02-08 01:03:25,282 INFO [Listener,port=46245] ipc.SimpleRpcServer$Listener(258): Listener,port=46245: stopping 2023-02-08 01:03:25,283 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(158): RpcServer.responder: stopped 2023-02-08 01:03:25,283 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(62): RpcServer.responder: stopping 2023-02-08 01:03:25,293 INFO [Time-limited test] hbase.ResourceChecker(175): after: ipc.TestBlockingIPC#testAsyncTimeout Thread=16 (was 14) Potentially hanging thread: RpcServer.responder java.lang.Thread.isInterrupted(Native Method) java.lang.Thread.interrupted(Thread.java:952) org.apache.maven.surefire.booter.spi.EventChannelEncoder.write(EventChannelEncoder.java:304) org.apache.maven.surefire.booter.spi.EventChannelEncoder.setOutErr(EventChannelEncoder.java:189) org.apache.maven.surefire.booter.spi.EventChannelEncoder.testOutput(EventChannelEncoder.java:183) org.apache.maven.surefire.api.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.java:113) org.apache.maven.surefire.api.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.java:44) org.apache.maven.surefire.common.junit4.JUnit4RunListener.writeTestOutput(JUnit4RunListener.java:235) org.apache.maven.surefire.api.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleOutputCapture.java:64) sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291) sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295) sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141) java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229) org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:54) org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:317) org.apache.log4j.WriterAppender.append(WriterAppender.java:156) org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:232) org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:57) org.apache.log4j.Category.callAppenders(Category.java:206) org.apache.log4j.Category.forcedLog(Category.java:384) org.apache.log4j.Category.log(Category.java:810) org.slf4j.impl.Reload4jLoggerAdapter.info(Reload4jLoggerAdapter.java:284) org.apache.hadoop.hbase.ipc.SimpleRpcServerResponder.doRunLoop(SimpleRpcServerResponder.java:158) org.apache.hadoop.hbase.ipc.SimpleRpcServerResponder.run(SimpleRpcServerResponder.java:60) - Thread LEAK? -, OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=546 (was 546), ProcessCount=174 (was 174), AvailableMemoryMB=4032 (was 4026) - AvailableMemoryMB LEAK? -