2024-02-07 10:55:16,654 DEBUG [main] util.ClassSize(228): Using Unsafe to estimate memory layout 2024-02-07 10:55:16,791 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.ipc.TestBlockingIPC timeout: 13 mins 2024-02-07 10:55:16,825 INFO [Time-limited test] hbase.ResourceChecker(147): before: ipc.TestBlockingIPC#testRTEDuringConnectionSetup Thread=10, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=177, ProcessCount=168, AvailableMemoryMB=3669 2024-02-07 10:55:16,985 INFO [Time-limited test] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.SimpleRpcServer hosting hbase.test.pb.TestProtobufRpcProto 2024-02-07 10:55:17,189 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-02-07 10:55:17,288 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-02-07 10:55:17,327 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2024-02-07 10:55:17,434 INFO [Time-limited test] ipc.SimpleRpcServer$Listener(134): testRpcServer: started 3 reader(s) listening on port=44273 2024-02-07 10:55:17,534 DEBUG [Time-limited test] ipc.AbstractRpcClient(189): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3bed8587, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2024-02-07 10:55:17,543 INFO [Time-limited test] ipc.FifoRpcScheduler(61): Using FifoRpcScheduler as user call queue; handlerCount=1; maxQueueLength=10 2024-02-07 10:55:17,543 INFO [Listener,port=44273] ipc.SimpleRpcServer$Listener(214): Listener,port=44273: starting 2024-02-07 10:55:17,543 DEBUG [RpcServer.responder] ipc.SimpleRpcServerResponder(58): RpcServer.responder: starting 2024-02-07 10:55:17,831 DEBUG [Time-limited test] ipc.RpcConnection(122): Using SIMPLE authentication for service=TestProtobufRpcProto, sasl=false 2024-02-07 10:55:17,910 DEBUG [Time-limited test] ipc.BlockingRpcConnection(501): Connecting to localhost:44273 2024-02-07 10:55:17,913 DEBUG [Time-limited test] ipc.FailedServers(52): Added failed server with address localhost:44273 to list caused by java.lang.RuntimeException: Injected fault 2024-02-07 10:55:17,913 DEBUG [Time-limited test] ipc.AbstractRpcClient(489): Stopping rpc client 2024-02-07 10:55:17,914 INFO [Time-limited test] ipc.AbstractTestIPC(171): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.io.IOException: Call to address=localhost:44273 failed on local exception: java.io.IOException: Could not set up IO Streams to localhost:44273 2024-02-07 10:55:17,914 INFO [Time-limited test] ipc.SimpleRpcServer(438): Stopping server on 44273 2024-02-07 10:55:17,915 INFO [Listener,port=44273] ipc.SimpleRpcServer$Listener(258): Listener,port=44273: stopping 2024-02-07 10:55:17,915 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(158): RpcServer.responder: stopped 2024-02-07 10:55:17,915 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(62): RpcServer.responder: stopping 2024-02-07 10:55:17,923 INFO [Time-limited test] hbase.ResourceChecker(175): after: ipc.TestBlockingIPC#testRTEDuringConnectionSetup Thread=13 (was 10) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: HBase-Metrics2-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Idle-Rpc-Conn-Sweeper-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=291 (was 290) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=194 (was 177) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=3393 (was 3669) 2024-02-07 10:55:17,932 INFO [Time-limited test] hbase.ResourceChecker(147): before: ipc.TestBlockingIPC#testTimeout Thread=13, OpenFileDescriptor=291, MaxFileDescriptor=60000, SystemLoadAverage=194, ProcessCount=168, AvailableMemoryMB=3392 2024-02-07 10:55:17,933 INFO [Time-limited test] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.SimpleRpcServer hosting hbase.test.pb.TestProtobufRpcProto 2024-02-07 10:55:17,934 INFO [Time-limited test] ipc.SimpleRpcServer$Listener(134): testRpcServer: started 3 reader(s) listening on port=40197 2024-02-07 10:55:17,935 DEBUG [Time-limited test] ipc.AbstractRpcClient(189): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@60889613, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2024-02-07 10:55:17,935 DEBUG [RpcServer.responder] ipc.SimpleRpcServerResponder(58): RpcServer.responder: starting 2024-02-07 10:55:17,935 INFO [Listener,port=40197] ipc.SimpleRpcServer$Listener(214): Listener,port=40197: starting 2024-02-07 10:55:17,935 INFO [Time-limited test] ipc.FifoRpcScheduler(61): Using FifoRpcScheduler as user call queue; handlerCount=1; maxQueueLength=10 2024-02-07 10:55:17,938 DEBUG [Time-limited test] ipc.RpcConnection(122): Using SIMPLE authentication for service=TestProtobufRpcProto, sasl=false 2024-02-07 10:55:17,940 DEBUG [Time-limited test] ipc.BlockingRpcConnection(501): Connecting to localhost:40197 2024-02-07 10:55:17,965 INFO [Reader=1,bindAddress=localhost,port=40197] ipc.ServerRpcConnection(540): Connection from 127.0.0.1:58714, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=TestProtobufRpcProto 2024-02-07 10:55:18,058 INFO [Time-limited test] ipc.AbstractTestIPC(293): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost:40197 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=0,methodName=pause], waitTime=120ms, rpcTimeout=100ms 2024-02-07 10:55:18,160 INFO [Time-limited test] ipc.AbstractTestIPC(293): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost:40197 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=1,methodName=pause], waitTime=102ms, rpcTimeout=100ms 2024-02-07 10:55:18,270 INFO [Time-limited test] ipc.AbstractTestIPC(293): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost:40197 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=2,methodName=pause], waitTime=109ms, rpcTimeout=100ms 2024-02-07 10:55:18,380 INFO [Time-limited test] ipc.AbstractTestIPC(293): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost:40197 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=3,methodName=pause], waitTime=110ms, rpcTimeout=100ms 2024-02-07 10:55:18,490 INFO [Time-limited test] ipc.AbstractTestIPC(293): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost:40197 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=4,methodName=pause], waitTime=110ms, rpcTimeout=100ms 2024-02-07 10:55:18,600 INFO [Time-limited test] ipc.AbstractTestIPC(293): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost:40197 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=5,methodName=pause], waitTime=110ms, rpcTimeout=100ms 2024-02-07 10:55:18,710 INFO [Time-limited test] ipc.AbstractTestIPC(293): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost:40197 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=6,methodName=pause], waitTime=109ms, rpcTimeout=100ms 2024-02-07 10:55:18,820 INFO [Time-limited test] ipc.AbstractTestIPC(293): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost:40197 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=7,methodName=pause], waitTime=109ms, rpcTimeout=100ms 2024-02-07 10:55:18,930 INFO [Time-limited test] ipc.AbstractTestIPC(293): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost:40197 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=8,methodName=pause], waitTime=109ms, rpcTimeout=100ms 2024-02-07 10:55:18,997 DEBUG [BRPC Connection (681381511) to localhost:40197 from jenkins (attempt: 1)] ipc.BlockingRpcConnection(741): Unknown callId: 0, skipping over this response of 1 bytes 2024-02-07 10:55:19,008 WARN [FifoRpcScheduler.handler-pool-0] ipc.CallRunner(105): Dropping timed out call: callId: 1 service: TestProtobufRpcProto methodName: pause size: 20 connection: 127.0.0.1:58714 deadline: 1707303318159 param: TODO: class org.apache.hadoop.hbase.shaded.ipc.protobuf.generated.TestProtos$PauseRequestProto connection: 127.0.0.1:58714 2024-02-07 10:55:19,009 WARN [FifoRpcScheduler.handler-pool-0] ipc.CallRunner(105): Dropping timed out call: callId: 2 service: TestProtobufRpcProto methodName: pause size: 20 connection: 127.0.0.1:58714 deadline: 1707303318261 param: TODO: class org.apache.hadoop.hbase.shaded.ipc.protobuf.generated.TestProtos$PauseRequestProto connection: 127.0.0.1:58714 2024-02-07 10:55:19,009 WARN [FifoRpcScheduler.handler-pool-0] ipc.CallRunner(105): Dropping timed out call: callId: 3 service: TestProtobufRpcProto methodName: pause size: 20 connection: 127.0.0.1:58714 deadline: 1707303318371 param: TODO: class org.apache.hadoop.hbase.shaded.ipc.protobuf.generated.TestProtos$PauseRequestProto connection: 127.0.0.1:58714 2024-02-07 10:55:19,009 WARN [FifoRpcScheduler.handler-pool-0] ipc.CallRunner(105): Dropping timed out call: callId: 4 service: TestProtobufRpcProto methodName: pause size: 20 connection: 127.0.0.1:58714 deadline: 1707303318480 param: TODO: class org.apache.hadoop.hbase.shaded.ipc.protobuf.generated.TestProtos$PauseRequestProto connection: 127.0.0.1:58714 2024-02-07 10:55:19,009 WARN [FifoRpcScheduler.handler-pool-0] ipc.CallRunner(105): Dropping timed out call: callId: 5 service: TestProtobufRpcProto methodName: pause size: 20 connection: 127.0.0.1:58714 deadline: 1707303318590 param: TODO: class org.apache.hadoop.hbase.shaded.ipc.protobuf.generated.TestProtos$PauseRequestProto connection: 127.0.0.1:58714 2024-02-07 10:55:19,009 WARN [FifoRpcScheduler.handler-pool-0] ipc.CallRunner(105): Dropping timed out call: callId: 6 service: TestProtobufRpcProto methodName: pause size: 20 connection: 127.0.0.1:58714 deadline: 1707303318701 param: TODO: class org.apache.hadoop.hbase.shaded.ipc.protobuf.generated.TestProtos$PauseRequestProto connection: 127.0.0.1:58714 2024-02-07 10:55:19,009 WARN [FifoRpcScheduler.handler-pool-0] ipc.CallRunner(105): Dropping timed out call: callId: 7 service: TestProtobufRpcProto methodName: pause size: 20 connection: 127.0.0.1:58714 deadline: 1707303318811 param: TODO: class org.apache.hadoop.hbase.shaded.ipc.protobuf.generated.TestProtos$PauseRequestProto connection: 127.0.0.1:58714 2024-02-07 10:55:19,010 WARN [FifoRpcScheduler.handler-pool-0] ipc.CallRunner(105): Dropping timed out call: callId: 8 service: TestProtobufRpcProto methodName: pause size: 20 connection: 127.0.0.1:58714 deadline: 1707303318921 param: TODO: class org.apache.hadoop.hbase.shaded.ipc.protobuf.generated.TestProtos$PauseRequestProto connection: 127.0.0.1:58714 2024-02-07 10:55:19,040 INFO [Time-limited test] ipc.AbstractTestIPC(293): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost:40197 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=9,methodName=pause], waitTime=109ms, rpcTimeout=100ms 2024-02-07 10:55:19,041 DEBUG [Time-limited test] ipc.AbstractRpcClient(489): Stopping rpc client 2024-02-07 10:55:19,041 INFO [Time-limited test] ipc.SimpleRpcServer(438): Stopping server on 40197 2024-02-07 10:55:19,041 INFO [Listener,port=40197] ipc.SimpleRpcServer$Listener(258): Listener,port=40197: stopping 2024-02-07 10:55:19,041 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(158): RpcServer.responder: stopped 2024-02-07 10:55:19,042 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(62): RpcServer.responder: stopping 2024-02-07 10:55:19,052 INFO [Time-limited test] hbase.ResourceChecker(175): after: ipc.TestBlockingIPC#testTimeout Thread=18 (was 13) Potentially hanging thread: FifoRpcScheduler.handler-pool-0 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.util.Threads.sleepWithoutInterrupt(Threads.java:152) org.apache.hadoop.hbase.ipc.TestProtobufRpcServiceImpl.pause(TestProtobufRpcServiceImpl.java:110) org.apache.hadoop.hbase.shaded.ipc.protobuf.generated.TestRpcServiceProtos$TestProtobufRpcProto$2.callBlockingMethod(TestRpcServiceProtos.java:144) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:111) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcClient-timer-pool-0 java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:600) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:496) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Reader=2,bindAddress=localhost,port=40197 sun.misc.Unsafe.getInt(Native Method) sun.nio.ch.NativeObject.getInt(NativeObject.java:257) sun.nio.ch.EPollArrayWrapper.getDescriptor(EPollArrayWrapper.java:165) sun.nio.ch.EPollSelectorImpl.updateSelectedKeys(EPollSelectorImpl.java:119) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:98) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener$Reader.doRunLoop(SimpleRpcServer.java:174) org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener$Reader.run(SimpleRpcServer.java:154) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.responder sun.nio.ch.FileDispatcherImpl.closeIntFD(Native Method) sun.nio.ch.EPollSelectorImpl.implClose(EPollSelectorImpl.java:150) sun.nio.ch.SelectorImpl.implCloseSelector(SelectorImpl.java:113) java.nio.channels.spi.AbstractSelector.close(AbstractSelector.java:111) org.apache.hadoop.hbase.ipc.SimpleRpcServerResponder.run(SimpleRpcServerResponder.java:64) Potentially hanging thread: Monitor thread for TaskMonitor java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.monitoring.TaskMonitor$MonitorRunnable.run(TaskMonitor.java:327) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=294 (was 291) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=194 (was 194), ProcessCount=168 (was 168), AvailableMemoryMB=3172 (was 3392) 2024-02-07 10:55:19,063 INFO [Time-limited test] hbase.ResourceChecker(147): before: ipc.TestBlockingIPC#testRpcScheduler Thread=16, OpenFileDescriptor=294, MaxFileDescriptor=60000, SystemLoadAverage=194, ProcessCount=168, AvailableMemoryMB=3172 2024-02-07 10:55:19,585 INFO [Time-limited test] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.SimpleRpcServer hosting hbase.test.pb.TestProtobufRpcProto 2024-02-07 10:55:19,587 INFO [Time-limited test] ipc.SimpleRpcServer$Listener(134): testRpcServer: started 3 reader(s) listening on port=42951 2024-02-07 10:55:19,613 DEBUG [Time-limited test] ipc.AbstractRpcClient(189): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a1da8f0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2024-02-07 10:55:19,613 DEBUG [RpcServer.responder] ipc.SimpleRpcServerResponder(58): RpcServer.responder: starting 2024-02-07 10:55:19,613 INFO [Time-limited test] ipc.FifoRpcScheduler(61): Using FifoRpcScheduler$MockitoMock$2126951374 as user call queue; handlerCount=1; maxQueueLength=10 2024-02-07 10:55:19,613 INFO [Listener,port=42951] ipc.SimpleRpcServer$Listener(214): Listener,port=42951: starting 2024-02-07 10:55:19,618 DEBUG [Time-limited test] ipc.RpcConnection(122): Using SIMPLE authentication for service=TestProtobufRpcProto, sasl=false 2024-02-07 10:55:19,619 DEBUG [Time-limited test] ipc.BlockingRpcConnection(501): Connecting to localhost:42951 2024-02-07 10:55:19,620 INFO [Reader=1,bindAddress=localhost,port=42951] ipc.ServerRpcConnection(540): Connection from 127.0.0.1:40454, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=TestProtobufRpcProto 2024-02-07 10:55:19,634 DEBUG [Time-limited test] ipc.AbstractRpcClient(489): Stopping rpc client 2024-02-07 10:55:19,634 INFO [Time-limited test] ipc.SimpleRpcServer(438): Stopping server on 42951 2024-02-07 10:55:19,634 DEBUG [BRPC Connection (681381511) to localhost:42951 from jenkins (attempt: 1)] ipc.BlockingRpcConnection(369): Interrupted while waiting for work 2024-02-07 10:55:19,634 INFO [Listener,port=42951] ipc.SimpleRpcServer$Listener(258): Listener,port=42951: stopping 2024-02-07 10:55:19,635 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(158): RpcServer.responder: stopped 2024-02-07 10:55:19,635 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(62): RpcServer.responder: stopping 2024-02-07 10:55:19,642 INFO [Time-limited test] hbase.ResourceChecker(175): after: ipc.TestBlockingIPC#testRpcScheduler Thread=17 (was 16) Potentially hanging thread: RpcServer.responder sun.nio.ch.FileDispatcherImpl.closeIntFD(Native Method) sun.nio.ch.EPollSelectorImpl.implClose(EPollSelectorImpl.java:150) sun.nio.ch.SelectorImpl.implCloseSelector(SelectorImpl.java:113) java.nio.channels.spi.AbstractSelector.close(AbstractSelector.java:111) org.apache.hadoop.hbase.ipc.SimpleRpcServerResponder.run(SimpleRpcServerResponder.java:64) - Thread LEAK? -, OpenFileDescriptor=294 (was 294), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=194 (was 194), ProcessCount=168 (was 168), AvailableMemoryMB=3053 (was 3172) 2024-02-07 10:55:19,649 INFO [Time-limited test] hbase.ResourceChecker(147): before: ipc.TestBlockingIPC#testAsyncRemoteError Thread=16, OpenFileDescriptor=294, MaxFileDescriptor=60000, SystemLoadAverage=194, ProcessCount=168, AvailableMemoryMB=3052 2024-02-07 10:55:19,650 DEBUG [Time-limited test] ipc.AbstractRpcClient(189): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a86b1c3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2024-02-07 10:55:19,650 INFO [Time-limited test] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.SimpleRpcServer hosting hbase.test.pb.TestProtobufRpcProto 2024-02-07 10:55:19,651 INFO [Time-limited test] ipc.SimpleRpcServer$Listener(134): testRpcServer: started 3 reader(s) listening on port=34951 2024-02-07 10:55:19,651 DEBUG [RpcServer.responder] ipc.SimpleRpcServerResponder(58): RpcServer.responder: starting 2024-02-07 10:55:19,651 INFO [Time-limited test] ipc.FifoRpcScheduler(61): Using FifoRpcScheduler as user call queue; handlerCount=1; maxQueueLength=10 2024-02-07 10:55:19,651 INFO [Listener,port=34951] ipc.SimpleRpcServer$Listener(214): Listener,port=34951: starting 2024-02-07 10:55:19,653 DEBUG [Time-limited test] ipc.RpcConnection(122): Using SIMPLE authentication for service=TestProtobufRpcProto, sasl=false 2024-02-07 10:55:19,654 DEBUG [Time-limited test] ipc.BlockingRpcConnection(501): Connecting to localhost:34951 2024-02-07 10:55:19,654 INFO [Reader=1,bindAddress=localhost,port=34951] ipc.ServerRpcConnection(540): Connection from 127.0.0.1:53898, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=TestProtobufRpcProto 2024-02-07 10:55:19,659 DEBUG [FifoRpcScheduler.handler-pool-0] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.DoNotRetryIOException: server error! at org.apache.hadoop.hbase.ipc.TestProtobufRpcServiceImpl.error(TestProtobufRpcServiceImpl.java:104) at org.apache.hadoop.hbase.shaded.ipc.protobuf.generated.TestRpcServiceProtos$TestProtobufRpcProto$2.callBlockingMethod(TestRpcServiceProtos.java:142) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:111) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-02-07 10:55:19,661 DEBUG [FifoRpcScheduler.handler-pool-0] ipc.CallRunner(144): callId: 0 service: TestProtobufRpcProto methodName: error size: 17 connection: 127.0.0.1:53898 deadline: 9223372036854775807, exception=org.apache.hadoop.hbase.DoNotRetryIOException: server error! 2024-02-07 10:55:19,664 INFO [Time-limited test] ipc.AbstractTestIPC(374): Caught expected exception: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.DoNotRetryIOException): org.apache.hadoop.hbase.DoNotRetryIOException: server error! at org.apache.hadoop.hbase.ipc.TestProtobufRpcServiceImpl.error(TestProtobufRpcServiceImpl.java:104) at org.apache.hadoop.hbase.shaded.ipc.protobuf.generated.TestRpcServiceProtos$TestProtobufRpcProto$2.callBlockingMethod(TestRpcServiceProtos.java:142) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:111) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-02-07 10:55:19,935 DEBUG [Time-limited test] ipc.AbstractRpcClient(489): Stopping rpc client 2024-02-07 10:55:19,935 INFO [Time-limited test] ipc.SimpleRpcServer(438): Stopping server on 34951 2024-02-07 10:55:19,936 INFO [Listener,port=34951] ipc.SimpleRpcServer$Listener(258): Listener,port=34951: stopping 2024-02-07 10:55:19,936 DEBUG [BRPC Connection (681381511) to localhost:34951 from jenkins (attempt: 1)] ipc.BlockingRpcConnection(369): Interrupted while waiting for work 2024-02-07 10:55:19,936 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(158): RpcServer.responder: stopped 2024-02-07 10:55:19,937 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(62): RpcServer.responder: stopping 2024-02-07 10:55:19,946 INFO [Time-limited test] hbase.ResourceChecker(175): after: ipc.TestBlockingIPC#testAsyncRemoteError Thread=19 (was 16) Potentially hanging thread: BRPC Connection (681381511) to localhost:34951 from jenkins (attempt: 1) org.apache.log4j.Category.callAppenders(Category.java:211) org.apache.log4j.Category.forcedLog(Category.java:384) org.apache.log4j.Category.log(Category.java:810) org.slf4j.impl.Reload4jLoggerAdapter.debug(Reload4jLoggerAdapter.java:189) org.apache.hadoop.hbase.ipc.BlockingRpcConnection.waitForWork(BlockingRpcConnection.java:369) org.apache.hadoop.hbase.ipc.BlockingRpcConnection.run(BlockingRpcConnection.java:392) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3693) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.responder org.apache.log4j.Category.callAppenders(Category.java:205) org.apache.log4j.Category.forcedLog(Category.java:384) org.apache.log4j.Category.log(Category.java:810) org.slf4j.impl.Reload4jLoggerAdapter.info(Reload4jLoggerAdapter.java:284) org.apache.hadoop.hbase.ipc.SimpleRpcServerResponder.doRunLoop(SimpleRpcServerResponder.java:158) org.apache.hadoop.hbase.ipc.SimpleRpcServerResponder.run(SimpleRpcServerResponder.java:60) - Thread LEAK? -, OpenFileDescriptor=294 (was 294), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=194 (was 194), ProcessCount=168 (was 168), AvailableMemoryMB=2931 (was 3052) 2024-02-07 10:55:19,960 INFO [Time-limited test] hbase.ResourceChecker(147): before: ipc.TestBlockingIPC#testAsyncEcho Thread=17, OpenFileDescriptor=294, MaxFileDescriptor=60000, SystemLoadAverage=194, ProcessCount=168, AvailableMemoryMB=2929 2024-02-07 10:55:19,986 INFO [Time-limited test] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.SimpleRpcServer hosting hbase.test.pb.TestProtobufRpcProto 2024-02-07 10:55:19,988 INFO [Time-limited test] ipc.SimpleRpcServer$Listener(134): testRpcServer: started 3 reader(s) listening on port=39851 2024-02-07 10:55:19,989 DEBUG [Time-limited test] ipc.AbstractRpcClient(189): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39aad62d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2024-02-07 10:55:19,989 DEBUG [RpcServer.responder] ipc.SimpleRpcServerResponder(58): RpcServer.responder: starting 2024-02-07 10:55:19,990 INFO [Time-limited test] ipc.FifoRpcScheduler(61): Using FifoRpcScheduler as user call queue; handlerCount=1; maxQueueLength=10 2024-02-07 10:55:19,990 DEBUG [Time-limited test] ipc.RpcConnection(122): Using SIMPLE authentication for service=TestProtobufRpcProto, sasl=false 2024-02-07 10:55:19,991 DEBUG [Time-limited test] ipc.BlockingRpcConnection(501): Connecting to localhost:39851 2024-02-07 10:55:19,992 INFO [Listener,port=39851] ipc.SimpleRpcServer$Listener(214): Listener,port=39851: starting 2024-02-07 10:55:19,993 INFO [Reader=1,bindAddress=localhost,port=39851] ipc.ServerRpcConnection(540): Connection from 127.0.0.1:47226, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=TestProtobufRpcProto 2024-02-07 10:55:19,999 DEBUG [Time-limited test] ipc.AbstractRpcClient(489): Stopping rpc client 2024-02-07 10:55:20,000 INFO [Time-limited test] ipc.SimpleRpcServer(438): Stopping server on 39851 2024-02-07 10:55:20,000 DEBUG [BRPC Connection (681381511) to localhost:39851 from jenkins (attempt: 1)] ipc.BlockingRpcConnection(369): Interrupted while waiting for work 2024-02-07 10:55:20,000 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(158): RpcServer.responder: stopped 2024-02-07 10:55:20,001 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(62): RpcServer.responder: stopping 2024-02-07 10:55:20,000 INFO [Listener,port=39851] ipc.SimpleRpcServer$Listener(258): Listener,port=39851: stopping 2024-02-07 10:55:20,010 INFO [Time-limited test] hbase.ResourceChecker(175): after: ipc.TestBlockingIPC#testAsyncEcho Thread=21 (was 17) Potentially hanging thread: RpcServer.responder java.lang.Throwable.getStackTraceElement(Native Method) java.lang.Throwable.getOurStackTrace(Throwable.java:828) java.lang.Throwable.getStackTrace(Throwable.java:817) org.apache.log4j.spi.LocationInfo.(LocationInfo.java:98) org.apache.log4j.spi.LoggingEvent.getLocationInformation(LoggingEvent.java:260) org.apache.log4j.helpers.PatternParser$ClassNamePatternConverter.getFullyQualifiedName(PatternParser.java:518) org.apache.log4j.helpers.PatternParser$NamedPatternConverter.convert(PatternParser.java:491) org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:63) org.apache.log4j.PatternLayout.format(PatternLayout.java:510) org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:303) org.apache.log4j.WriterAppender.append(WriterAppender.java:156) org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:232) org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:57) org.apache.log4j.Category.callAppenders(Category.java:206) org.apache.log4j.Category.forcedLog(Category.java:384) org.apache.log4j.Category.log(Category.java:810) org.slf4j.impl.Reload4jLoggerAdapter.info(Reload4jLoggerAdapter.java:284) org.apache.hadoop.hbase.ipc.SimpleRpcServerResponder.doRunLoop(SimpleRpcServerResponder.java:158) org.apache.hadoop.hbase.ipc.SimpleRpcServerResponder.run(SimpleRpcServerResponder.java:60) Potentially hanging thread: Listener,port=39851 org.apache.log4j.Category.callAppenders(Category.java:205) org.apache.log4j.Category.forcedLog(Category.java:384) org.apache.log4j.Category.log(Category.java:810) org.slf4j.impl.Reload4jLoggerAdapter.info(Reload4jLoggerAdapter.java:284) org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener.run(SimpleRpcServer.java:258) Potentially hanging thread: RpcServer idle connection scanner for port 39851 java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) - Thread LEAK? -, OpenFileDescriptor=294 (was 294), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=194 (was 194), ProcessCount=168 (was 168), AvailableMemoryMB=2924 (was 2929) 2024-02-07 10:55:20,010 DEBUG [FifoRpcScheduler.handler-pool-0] ipc.SimpleRpcServerResponder(238): 127.0.0.1:58714: output error -- closing 2024-02-07 10:55:20,010 WARN [FifoRpcScheduler.handler-pool-0] ipc.CallRunner(181): FifoRpcScheduler.handler-pool-0: caught a ClosedChannelException, this means that the server localhost/127.0.0.1:40197 was processing a request but the client went away. The error message was: null 2024-02-07 10:55:20,019 INFO [Time-limited test] hbase.ResourceChecker(147): before: ipc.TestBlockingIPC#testConnectionCloseWithOutstandingRPCs Thread=17, OpenFileDescriptor=294, MaxFileDescriptor=60000, SystemLoadAverage=194, ProcessCount=168, AvailableMemoryMB=2921 2024-02-07 10:55:20,024 INFO [Time-limited test] ipc.SimpleRpcServer$Listener(134): testRpcServer: started 3 reader(s) listening on port=41811 2024-02-07 10:55:20,025 DEBUG [Time-limited test] ipc.AbstractRpcClient(189): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49d6d526, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2024-02-07 10:55:20,029 DEBUG [RpcServer.responder] ipc.SimpleRpcServerResponder(58): RpcServer.responder: starting 2024-02-07 10:55:20,029 INFO [Listener,port=41811] ipc.SimpleRpcServer$Listener(214): Listener,port=41811: starting 2024-02-07 10:55:20,029 INFO [Time-limited test] ipc.FifoRpcScheduler(61): Using FifoRpcScheduler as user call queue; handlerCount=1; maxQueueLength=10 2024-02-07 10:55:20,029 DEBUG [Time-limited test] ipc.RpcConnection(122): Using SIMPLE authentication for service=TestProtobufRpcProto, sasl=false 2024-02-07 10:55:20,030 DEBUG [Time-limited test] ipc.BlockingRpcConnection(501): Connecting to localhost:41811 2024-02-07 10:55:20,031 INFO [Reader=1,bindAddress=localhost,port=41811] ipc.ServerRpcConnection(540): Connection from 127.0.0.1:35764, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=TestProtobufRpcProto 2024-02-07 10:55:20,031 DEBUG [Reader=1,bindAddress=localhost,port=41811] ipc.SimpleRpcServer$Listener(326): Caught exception while reading: org.apache.hadoop.hbase.DoNotRetryIOException: Failing for test at org.apache.hadoop.hbase.ipc.TestBlockingIPC$TestFailingRpcServer$FailingConnection.processRequest(TestBlockingIPC.java:94) at org.apache.hadoop.hbase.ipc.ServerRpcConnection.processOneRpc(ServerRpcConnection.java:444) at org.apache.hadoop.hbase.ipc.SimpleServerRpcConnection.process(SimpleServerRpcConnection.java:286) at org.apache.hadoop.hbase.ipc.SimpleServerRpcConnection.readAndProcess(SimpleServerRpcConnection.java:252) at org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener.doRead(SimpleRpcServer.java:319) at org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener$Reader.doRunLoop(SimpleRpcServer.java:181) at org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener$Reader.run(SimpleRpcServer.java:154) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-02-07 10:55:20,032 DEBUG [BRPC Connection (681381511) to localhost:41811 from jenkins (attempt: 1)] ipc.BlockingRpcConnection(786): Closing connection after error in call null java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.hadoop.hbase.ipc.BlockingRpcConnection.readResponse(BlockingRpcConnection.java:707) at org.apache.hadoop.hbase.ipc.BlockingRpcConnection.run(BlockingRpcConnection.java:393) at java.lang.Thread.run(Thread.java:750) 2024-02-07 10:55:20,032 DEBUG [Time-limited test] ipc.AbstractRpcClient(489): Stopping rpc client 2024-02-07 10:55:20,032 INFO [Time-limited test] ipc.AbstractTestIPC(324): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.io.EOFException: Call to address=localhost:41811 failed on local exception: java.io.EOFException 2024-02-07 10:55:20,032 INFO [Time-limited test] ipc.SimpleRpcServer(438): Stopping server on 41811 2024-02-07 10:55:20,033 INFO [Listener,port=41811] ipc.SimpleRpcServer$Listener(258): Listener,port=41811: stopping 2024-02-07 10:55:20,033 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(158): RpcServer.responder: stopped 2024-02-07 10:55:20,033 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(62): RpcServer.responder: stopping 2024-02-07 10:55:20,042 INFO [Time-limited test] hbase.ResourceChecker(175): after: ipc.TestBlockingIPC#testConnectionCloseWithOutstandingRPCs Thread=19 (was 17) Potentially hanging thread: RpcServer.responder java.lang.Throwable.getStackTraceElement(Native Method) java.lang.Throwable.getOurStackTrace(Throwable.java:828) java.lang.Throwable.getStackTrace(Throwable.java:817) org.apache.log4j.spi.LocationInfo.(LocationInfo.java:98) org.apache.log4j.spi.LoggingEvent.getLocationInformation(LoggingEvent.java:260) org.apache.log4j.helpers.PatternParser$ClassNamePatternConverter.getFullyQualifiedName(PatternParser.java:518) org.apache.log4j.helpers.PatternParser$NamedPatternConverter.convert(PatternParser.java:491) org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:63) org.apache.log4j.PatternLayout.format(PatternLayout.java:510) org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:303) org.apache.log4j.WriterAppender.append(WriterAppender.java:156) org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:232) org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:57) org.apache.log4j.Category.callAppenders(Category.java:206) org.apache.log4j.Category.forcedLog(Category.java:384) org.apache.log4j.Category.log(Category.java:810) org.slf4j.impl.Reload4jLoggerAdapter.info(Reload4jLoggerAdapter.java:284) org.apache.hadoop.hbase.ipc.SimpleRpcServerResponder.doRunLoop(SimpleRpcServerResponder.java:158) org.apache.hadoop.hbase.ipc.SimpleRpcServerResponder.run(SimpleRpcServerResponder.java:60) Potentially hanging thread: Listener,port=41811 Potentially hanging thread: RpcServer idle connection scanner for port 41811 java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) - Thread LEAK? -, OpenFileDescriptor=294 (was 294), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=194 (was 194), ProcessCount=168 (was 168), AvailableMemoryMB=2917 (was 2921) 2024-02-07 10:55:20,050 INFO [Time-limited test] hbase.ResourceChecker(147): before: ipc.TestBlockingIPC#testBadPreambleHeader Thread=16, OpenFileDescriptor=294, MaxFileDescriptor=60000, SystemLoadAverage=194, ProcessCount=168, AvailableMemoryMB=2914 2024-02-07 10:55:20,051 INFO [Time-limited test] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.SimpleRpcServer hosting 2024-02-07 10:55:20,052 INFO [Time-limited test] ipc.SimpleRpcServer$Listener(134): testRpcServer: started 3 reader(s) listening on port=36041 2024-02-07 10:55:20,053 DEBUG [Time-limited test] ipc.AbstractRpcClient(189): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@57a4ac4a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2024-02-07 10:55:20,053 INFO [Time-limited test] ipc.FifoRpcScheduler(61): Using FifoRpcScheduler as user call queue; handlerCount=1; maxQueueLength=10 2024-02-07 10:55:20,054 INFO [Listener,port=36041] ipc.SimpleRpcServer$Listener(214): Listener,port=36041: starting 2024-02-07 10:55:20,053 DEBUG [RpcServer.responder] ipc.SimpleRpcServerResponder(58): RpcServer.responder: starting 2024-02-07 10:55:20,055 DEBUG [Time-limited test] ipc.RpcConnection(122): Using SIMPLE authentication for service=TestProtobufRpcProto, sasl=false 2024-02-07 10:55:20,055 DEBUG [Time-limited test] ipc.BlockingRpcConnection(501): Connecting to localhost:36041 2024-02-07 10:55:20,056 WARN [Reader=1,bindAddress=localhost,port=36041] ipc.ServerRpcConnection(706): serverVersion=0, clientVersion=0, authMethod=-10, authName=unknown from 127.0.0.1:48656 2024-02-07 10:55:20,062 DEBUG [Time-limited test] ipc.AbstractRpcClient(489): Stopping rpc client 2024-02-07 10:55:20,062 INFO [Time-limited test] ipc.SimpleRpcServer(438): Stopping server on 36041 2024-02-07 10:55:20,063 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(158): RpcServer.responder: stopped 2024-02-07 10:55:20,063 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(62): RpcServer.responder: stopping 2024-02-07 10:55:20,063 INFO [Listener,port=36041] ipc.SimpleRpcServer$Listener(258): Listener,port=36041: stopping 2024-02-07 10:55:20,071 INFO [Time-limited test] hbase.ResourceChecker(175): after: ipc.TestBlockingIPC#testBadPreambleHeader Thread=19 (was 16) Potentially hanging thread: Listener,port=36041 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener.run(SimpleRpcServer.java:219) Potentially hanging thread: RpcServer.responder org.apache.maven.surefire.booter.spi.EventChannelEncoder.encodeMessage(EventChannelEncoder.java:378) org.apache.maven.surefire.booter.spi.EventChannelEncoder.setOutErr(EventChannelEncoder.java:172) org.apache.maven.surefire.booter.spi.EventChannelEncoder.testOutput(EventChannelEncoder.java:168) org.apache.maven.surefire.api.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.java:99) org.apache.maven.surefire.api.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.java:43) org.apache.maven.surefire.common.junit4.JUnit4RunListener.writeTestOutput(JUnit4RunListener.java:202) org.apache.maven.surefire.api.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleOutputCapture.java:57) sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291) sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295) sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141) java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229) org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:54) org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:317) org.apache.log4j.WriterAppender.append(WriterAppender.java:156) org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:232) org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:57) org.apache.log4j.Category.callAppenders(Category.java:206) org.apache.log4j.Category.forcedLog(Category.java:384) org.apache.log4j.Category.log(Category.java:810) org.slf4j.impl.Reload4jLoggerAdapter.info(Reload4jLoggerAdapter.java:284) org.apache.hadoop.hbase.ipc.SimpleRpcServerResponder.run(SimpleRpcServerResponder.java:62) Potentially hanging thread: RpcServer idle connection scanner for port 36041 java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) - Thread LEAK? -, OpenFileDescriptor=294 (was 294), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=194 (was 194), ProcessCount=168 (was 168), AvailableMemoryMB=2911 (was 2914) 2024-02-07 10:55:20,078 INFO [Time-limited test] hbase.ResourceChecker(147): before: ipc.TestBlockingIPC#testCompressCellBlock Thread=16, OpenFileDescriptor=294, MaxFileDescriptor=60000, SystemLoadAverage=194, ProcessCount=168, AvailableMemoryMB=2910 2024-02-07 10:55:20,091 INFO [Time-limited test] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.SimpleRpcServer hosting hbase.test.pb.TestProtobufRpcProto 2024-02-07 10:55:20,092 INFO [Time-limited test] ipc.SimpleRpcServer$Listener(134): testRpcServer: started 3 reader(s) listening on port=38927 2024-02-07 10:55:20,095 DEBUG [Time-limited test] ipc.AbstractRpcClient(189): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6e3f63c0, compressor=org.apache.hadoop.io.compress.GzipCodec@695337b, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2024-02-07 10:55:20,096 INFO [Time-limited test] ipc.FifoRpcScheduler(61): Using FifoRpcScheduler as user call queue; handlerCount=1; maxQueueLength=10 2024-02-07 10:55:20,096 INFO [Listener,port=38927] ipc.SimpleRpcServer$Listener(214): Listener,port=38927: starting 2024-02-07 10:55:20,096 DEBUG [RpcServer.responder] ipc.SimpleRpcServerResponder(58): RpcServer.responder: starting 2024-02-07 10:55:20,097 DEBUG [Time-limited test] ipc.RpcConnection(122): Using SIMPLE authentication for service=TestProtobufRpcProto, sasl=false 2024-02-07 10:55:20,125 DEBUG [Time-limited test] ipc.BlockingRpcConnection(501): Connecting to localhost:38927 2024-02-07 10:55:20,140 INFO [Reader=1,bindAddress=localhost,port=38927] ipc.ServerRpcConnection(540): Connection from 127.0.0.1:53422, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=TestProtobufRpcProto 2024-02-07 10:55:20,165 DEBUG [Time-limited test] ipc.AbstractRpcClient(489): Stopping rpc client 2024-02-07 10:55:20,166 INFO [Time-limited test] ipc.SimpleRpcServer(438): Stopping server on 38927 2024-02-07 10:55:20,166 DEBUG [BRPC Connection (681381511) to localhost:38927 from jenkins (attempt: 1)] ipc.BlockingRpcConnection(369): Interrupted while waiting for work 2024-02-07 10:55:20,166 INFO [Listener,port=38927] ipc.SimpleRpcServer$Listener(258): Listener,port=38927: stopping 2024-02-07 10:55:20,166 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(158): RpcServer.responder: stopped 2024-02-07 10:55:20,167 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(62): RpcServer.responder: stopping 2024-02-07 10:55:20,174 INFO [Time-limited test] hbase.ResourceChecker(175): after: ipc.TestBlockingIPC#testCompressCellBlock Thread=18 (was 16) Potentially hanging thread: RpcServer.responder org.apache.maven.surefire.booter.spi.EventChannelEncoder.encodeMessage(EventChannelEncoder.java:376) org.apache.maven.surefire.booter.spi.EventChannelEncoder.setOutErr(EventChannelEncoder.java:172) org.apache.maven.surefire.booter.spi.EventChannelEncoder.testOutput(EventChannelEncoder.java:168) org.apache.maven.surefire.api.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.java:99) org.apache.maven.surefire.api.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.java:43) org.apache.maven.surefire.common.junit4.JUnit4RunListener.writeTestOutput(JUnit4RunListener.java:202) org.apache.maven.surefire.api.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleOutputCapture.java:57) sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291) sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295) sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141) java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229) org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:54) org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:317) org.apache.log4j.WriterAppender.append(WriterAppender.java:156) org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:232) org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:57) org.apache.log4j.Category.callAppenders(Category.java:206) org.apache.log4j.Category.forcedLog(Category.java:384) org.apache.log4j.Category.log(Category.java:810) org.slf4j.impl.Reload4jLoggerAdapter.info(Reload4jLoggerAdapter.java:284) org.apache.hadoop.hbase.ipc.SimpleRpcServerResponder.doRunLoop(SimpleRpcServerResponder.java:158) org.apache.hadoop.hbase.ipc.SimpleRpcServerResponder.run(SimpleRpcServerResponder.java:60) Potentially hanging thread: RpcServer idle connection scanner for port 38927 java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) - Thread LEAK? -, OpenFileDescriptor=294 (was 294), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=194 (was 194), ProcessCount=168 (was 168), AvailableMemoryMB=2881 (was 2910) 2024-02-07 10:55:20,181 INFO [Time-limited test] hbase.ResourceChecker(147): before: ipc.TestBlockingIPC#testRpcServerForNotNullRemoteAddressInCallObject Thread=16, OpenFileDescriptor=294, MaxFileDescriptor=60000, SystemLoadAverage=194, ProcessCount=168, AvailableMemoryMB=2880 2024-02-07 10:55:20,181 INFO [Time-limited test] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.SimpleRpcServer hosting hbase.test.pb.TestProtobufRpcProto 2024-02-07 10:55:20,182 INFO [Time-limited test] ipc.SimpleRpcServer$Listener(134): testRpcServer: started 3 reader(s) listening on port=42781 2024-02-07 10:55:20,183 DEBUG [Time-limited test] ipc.AbstractRpcClient(189): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2f022e59, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2024-02-07 10:55:20,183 DEBUG [RpcServer.responder] ipc.SimpleRpcServerResponder(58): RpcServer.responder: starting 2024-02-07 10:55:20,183 INFO [Time-limited test] ipc.FifoRpcScheduler(61): Using FifoRpcScheduler as user call queue; handlerCount=1; maxQueueLength=10 2024-02-07 10:55:20,183 INFO [Listener,port=42781] ipc.SimpleRpcServer$Listener(214): Listener,port=42781: starting 2024-02-07 10:55:20,186 DEBUG [Time-limited test] ipc.RpcConnection(122): Using SIMPLE authentication for service=TestProtobufRpcProto, sasl=false 2024-02-07 10:55:20,187 DEBUG [Time-limited test] ipc.BlockingRpcConnection(501): Connecting to localhost:42781 2024-02-07 10:55:20,188 INFO [Reader=1,bindAddress=localhost,port=42781] ipc.ServerRpcConnection(540): Connection from 127.0.0.1:60640, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=TestProtobufRpcProto 2024-02-07 10:55:20,191 DEBUG [Time-limited test] ipc.AbstractRpcClient(489): Stopping rpc client 2024-02-07 10:55:20,191 INFO [Time-limited test] ipc.SimpleRpcServer(438): Stopping server on 42781 2024-02-07 10:55:20,191 DEBUG [BRPC Connection (681381511) to localhost:42781 from jenkins (attempt: 1)] ipc.BlockingRpcConnection(369): Interrupted while waiting for work 2024-02-07 10:55:20,192 INFO [Listener,port=42781] ipc.SimpleRpcServer$Listener(258): Listener,port=42781: stopping 2024-02-07 10:55:20,192 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(158): RpcServer.responder: stopped 2024-02-07 10:55:20,192 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(62): RpcServer.responder: stopping 2024-02-07 10:55:20,199 INFO [Time-limited test] hbase.ResourceChecker(175): after: ipc.TestBlockingIPC#testRpcServerForNotNullRemoteAddressInCallObject Thread=19 (was 16) Potentially hanging thread: RpcServer idle connection scanner for port 42781 java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RpcServer.responder org.apache.log4j.Category.callAppenders(Category.java:205) org.apache.log4j.Category.forcedLog(Category.java:384) org.apache.log4j.Category.log(Category.java:810) org.slf4j.impl.Reload4jLoggerAdapter.info(Reload4jLoggerAdapter.java:284) org.apache.hadoop.hbase.ipc.SimpleRpcServerResponder.doRunLoop(SimpleRpcServerResponder.java:158) org.apache.hadoop.hbase.ipc.SimpleRpcServerResponder.run(SimpleRpcServerResponder.java:60) Potentially hanging thread: Listener,port=42781 java.io.BufferedOutputStream.write(BufferedOutputStream.java:117) org.apache.maven.surefire.api.util.internal.Channels$4.writeImpl(Channels.java:199) org.apache.maven.surefire.api.util.internal.AbstractNoninterruptibleWritableChannel.write(AbstractNoninterruptibleWritableChannel.java:66) org.apache.maven.surefire.api.util.internal.AbstractNoninterruptibleWritableChannel.writeBuffered(AbstractNoninterruptibleWritableChannel.java:49) org.apache.maven.surefire.api.stream.AbstractStreamEncoder.write(AbstractStreamEncoder.java:69) org.apache.maven.surefire.booter.spi.EventChannelEncoder.write(EventChannelEncoder.java:288) org.apache.maven.surefire.booter.spi.EventChannelEncoder.setOutErr(EventChannelEncoder.java:173) org.apache.maven.surefire.booter.spi.EventChannelEncoder.testOutput(EventChannelEncoder.java:168) org.apache.maven.surefire.api.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.java:99) org.apache.maven.surefire.api.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.java:43) org.apache.maven.surefire.common.junit4.JUnit4RunListener.writeTestOutput(JUnit4RunListener.java:202) org.apache.maven.surefire.api.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleOutputCapture.java:57) sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291) sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295) sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141) java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229) org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:54) org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:317) org.apache.log4j.WriterAppender.append(WriterAppender.java:156) org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:232) org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:57) org.apache.log4j.Category.callAppenders(Category.java:206) org.apache.log4j.Category.forcedLog(Category.java:384) org.apache.log4j.Category.log(Category.java:810) org.slf4j.impl.Reload4jLoggerAdapter.info(Reload4jLoggerAdapter.java:284) org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener.run(SimpleRpcServer.java:258) - Thread LEAK? -, OpenFileDescriptor=294 (was 294), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=194 (was 194), ProcessCount=168 (was 168), AvailableMemoryMB=2879 (was 2880) 2024-02-07 10:55:20,206 INFO [Time-limited test] hbase.ResourceChecker(147): before: ipc.TestBlockingIPC#testRemoteError Thread=16, OpenFileDescriptor=294, MaxFileDescriptor=60000, SystemLoadAverage=194, ProcessCount=168, AvailableMemoryMB=2878 2024-02-07 10:55:20,206 INFO [Time-limited test] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.SimpleRpcServer hosting hbase.test.pb.TestProtobufRpcProto 2024-02-07 10:55:20,207 INFO [Time-limited test] ipc.SimpleRpcServer$Listener(134): testRpcServer: started 3 reader(s) listening on port=35715 2024-02-07 10:55:20,208 DEBUG [Time-limited test] ipc.AbstractRpcClient(189): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@380e0dc6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2024-02-07 10:55:20,208 DEBUG [RpcServer.responder] ipc.SimpleRpcServerResponder(58): RpcServer.responder: starting 2024-02-07 10:55:20,208 INFO [Listener,port=35715] ipc.SimpleRpcServer$Listener(214): Listener,port=35715: starting 2024-02-07 10:55:20,208 INFO [Time-limited test] ipc.FifoRpcScheduler(61): Using FifoRpcScheduler as user call queue; handlerCount=1; maxQueueLength=10 2024-02-07 10:55:20,209 DEBUG [Time-limited test] ipc.RpcConnection(122): Using SIMPLE authentication for service=TestProtobufRpcProto, sasl=false 2024-02-07 10:55:20,209 DEBUG [Time-limited test] ipc.BlockingRpcConnection(501): Connecting to localhost:35715 2024-02-07 10:55:20,211 INFO [Reader=1,bindAddress=localhost,port=35715] ipc.ServerRpcConnection(540): Connection from 127.0.0.1:49676, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=TestProtobufRpcProto 2024-02-07 10:55:20,211 DEBUG [FifoRpcScheduler.handler-pool-0] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.DoNotRetryIOException: server error! at org.apache.hadoop.hbase.ipc.TestProtobufRpcServiceImpl.error(TestProtobufRpcServiceImpl.java:104) at org.apache.hadoop.hbase.shaded.ipc.protobuf.generated.TestRpcServiceProtos$TestProtobufRpcProto$2.callBlockingMethod(TestRpcServiceProtos.java:142) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:111) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-02-07 10:55:20,212 DEBUG [FifoRpcScheduler.handler-pool-0] ipc.CallRunner(144): callId: 0 service: TestProtobufRpcProto methodName: error size: 17 connection: 127.0.0.1:49676 deadline: 9223372036854775807, exception=org.apache.hadoop.hbase.DoNotRetryIOException: server error! 2024-02-07 10:55:20,212 DEBUG [Time-limited test] ipc.AbstractRpcClient(489): Stopping rpc client 2024-02-07 10:55:20,213 INFO [Time-limited test] ipc.AbstractTestIPC(264): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.DoNotRetryIOException): org.apache.hadoop.hbase.DoNotRetryIOException: server error! at org.apache.hadoop.hbase.ipc.TestProtobufRpcServiceImpl.error(TestProtobufRpcServiceImpl.java:104) at org.apache.hadoop.hbase.shaded.ipc.protobuf.generated.TestRpcServiceProtos$TestProtobufRpcProto$2.callBlockingMethod(TestRpcServiceProtos.java:142) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:111) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-02-07 10:55:20,213 INFO [Time-limited test] ipc.SimpleRpcServer(438): Stopping server on 35715 2024-02-07 10:55:20,213 DEBUG [BRPC Connection (681381511) to localhost:35715 from jenkins (attempt: 1)] ipc.BlockingRpcConnection(369): Interrupted while waiting for work 2024-02-07 10:55:20,213 INFO [Listener,port=35715] ipc.SimpleRpcServer$Listener(258): Listener,port=35715: stopping 2024-02-07 10:55:20,214 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(158): RpcServer.responder: stopped 2024-02-07 10:55:20,215 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(62): RpcServer.responder: stopping 2024-02-07 10:55:20,222 INFO [Time-limited test] hbase.ResourceChecker(175): after: ipc.TestBlockingIPC#testRemoteError Thread=19 (was 16) Potentially hanging thread: Listener,port=35715 org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener.run(SimpleRpcServer.java:261) Potentially hanging thread: RpcServer.responder java.lang.Thread.blockedOn(Thread.java:241) java.lang.System$2.blockedOn(System.java:1267) java.nio.channels.spi.AbstractInterruptibleChannel.blockedOn(AbstractInterruptibleChannel.java:211) java.nio.channels.spi.AbstractSelector.end(AbstractSelector.java:231) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:95) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.hbase.ipc.SimpleRpcServerResponder.doRunLoop(SimpleRpcServerResponder.java:112) org.apache.hadoop.hbase.ipc.SimpleRpcServerResponder.run(SimpleRpcServerResponder.java:60) Potentially hanging thread: RpcServer idle connection scanner for port 35715 java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) - Thread LEAK? -, OpenFileDescriptor=294 (was 294), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=194 (was 194), ProcessCount=168 (was 168), AvailableMemoryMB=2876 (was 2878) 2024-02-07 10:55:20,229 INFO [Time-limited test] hbase.ResourceChecker(147): before: ipc.TestBlockingIPC#testNoCodec Thread=16, OpenFileDescriptor=294, MaxFileDescriptor=60000, SystemLoadAverage=194, ProcessCount=168, AvailableMemoryMB=2875 2024-02-07 10:55:20,245 INFO [Time-limited test] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.SimpleRpcServer hosting hbase.test.pb.TestProtobufRpcProto 2024-02-07 10:55:20,247 INFO [Time-limited test] ipc.SimpleRpcServer$Listener(134): testRpcServer: started 3 reader(s) listening on port=33055 2024-02-07 10:55:20,248 DEBUG [Time-limited test] ipc.AbstractRpcClient(189): Codec=null, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2024-02-07 10:55:20,248 INFO [Time-limited test] ipc.FifoRpcScheduler(61): Using FifoRpcScheduler as user call queue; handlerCount=1; maxQueueLength=10 2024-02-07 10:55:20,248 INFO [Listener,port=33055] ipc.SimpleRpcServer$Listener(214): Listener,port=33055: starting 2024-02-07 10:55:20,248 DEBUG [RpcServer.responder] ipc.SimpleRpcServerResponder(58): RpcServer.responder: starting 2024-02-07 10:55:20,249 DEBUG [Time-limited test] ipc.RpcConnection(122): Using SIMPLE authentication for service=TestProtobufRpcProto, sasl=false 2024-02-07 10:55:20,249 DEBUG [Time-limited test] ipc.BlockingRpcConnection(501): Connecting to localhost:33055 2024-02-07 10:55:20,251 INFO [Reader=1,bindAddress=localhost,port=33055] ipc.ServerRpcConnection(540): Connection from 127.0.0.1:59860, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=TestProtobufRpcProto 2024-02-07 10:55:20,253 DEBUG [Time-limited test] ipc.AbstractRpcClient(489): Stopping rpc client 2024-02-07 10:55:20,253 INFO [Time-limited test] ipc.SimpleRpcServer(438): Stopping server on 33055 2024-02-07 10:55:20,253 DEBUG [BRPC Connection (681381511) to localhost:33055 from jenkins (attempt: 1)] ipc.BlockingRpcConnection(369): Interrupted while waiting for work 2024-02-07 10:55:20,253 INFO [Listener,port=33055] ipc.SimpleRpcServer$Listener(258): Listener,port=33055: stopping 2024-02-07 10:55:20,257 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(158): RpcServer.responder: stopped 2024-02-07 10:55:20,257 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(62): RpcServer.responder: stopping 2024-02-07 10:55:20,266 INFO [Time-limited test] hbase.ResourceChecker(175): after: ipc.TestBlockingIPC#testNoCodec Thread=17 (was 16) - Thread LEAK? -, OpenFileDescriptor=294 (was 294), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=194 (was 194), ProcessCount=168 (was 168), AvailableMemoryMB=2878 (was 2875) - AvailableMemoryMB LEAK? - 2024-02-07 10:55:20,274 INFO [Time-limited test] hbase.ResourceChecker(147): before: ipc.TestBlockingIPC#testRpcMaxRequestSize Thread=16, OpenFileDescriptor=294, MaxFileDescriptor=60000, SystemLoadAverage=194, ProcessCount=168, AvailableMemoryMB=2875 2024-02-07 10:55:20,276 INFO [Time-limited test] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.SimpleRpcServer hosting hbase.test.pb.TestProtobufRpcProto 2024-02-07 10:55:20,277 INFO [Time-limited test] ipc.SimpleRpcServer$Listener(134): testRpcServer: started 3 reader(s) listening on port=46293 2024-02-07 10:55:20,278 DEBUG [Time-limited test] ipc.AbstractRpcClient(189): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5dc21389, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2024-02-07 10:55:20,278 DEBUG [RpcServer.responder] ipc.SimpleRpcServerResponder(58): RpcServer.responder: starting 2024-02-07 10:55:20,278 INFO [Time-limited test] ipc.FifoRpcScheduler(61): Using FifoRpcScheduler as user call queue; handlerCount=1; maxQueueLength=10 2024-02-07 10:55:20,278 INFO [Listener,port=46293] ipc.SimpleRpcServer$Listener(214): Listener,port=46293: starting 2024-02-07 10:55:20,288 DEBUG [Time-limited test] ipc.RpcConnection(122): Using SIMPLE authentication for service=TestProtobufRpcProto, sasl=false 2024-02-07 10:55:20,288 DEBUG [Time-limited test] ipc.BlockingRpcConnection(501): Connecting to localhost:46293 2024-02-07 10:55:20,289 INFO [Reader=1,bindAddress=localhost,port=46293] ipc.ServerRpcConnection(540): Connection from 127.0.0.1:58486, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=TestProtobufRpcProto 2024-02-07 10:55:20,289 WARN [Reader=1,bindAddress=localhost,port=46293] ipc.SimpleServerRpcConnection(187): RPC data length of 1260 received from 127.0.0.1 is greater than max allowed 1000. Set "hbase.ipc.max.request.size" on server to override this limit (not recommended) 2024-02-07 10:55:20,295 DEBUG [Time-limited test] ipc.AbstractRpcClient(489): Stopping rpc client 2024-02-07 10:55:20,295 INFO [Time-limited test] ipc.AbstractTestIPC(225): Caught expected exception: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.RequestTooBigException): RPC data length of 1260 received from 127.0.0.1 is greater than max allowed 1000. Set "hbase.ipc.max.request.size" on server to override this limit (not recommended) 2024-02-07 10:55:20,295 DEBUG [BRPC Connection (681381511) to localhost:46293 from jenkins (attempt: 1)] ipc.BlockingRpcConnection(369): Interrupted while waiting for work 2024-02-07 10:55:20,296 INFO [Time-limited test] ipc.SimpleRpcServer(438): Stopping server on 46293 2024-02-07 10:55:20,296 INFO [Listener,port=46293] ipc.SimpleRpcServer$Listener(258): Listener,port=46293: stopping 2024-02-07 10:55:20,296 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(158): RpcServer.responder: stopped 2024-02-07 10:55:20,296 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(62): RpcServer.responder: stopping 2024-02-07 10:55:20,305 INFO [Time-limited test] hbase.ResourceChecker(175): after: ipc.TestBlockingIPC#testRpcMaxRequestSize Thread=17 (was 16) - Thread LEAK? -, OpenFileDescriptor=294 (was 294), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=194 (was 194), ProcessCount=168 (was 168), AvailableMemoryMB=2869 (was 2875) 2024-02-07 10:55:20,312 INFO [Time-limited test] hbase.ResourceChecker(147): before: ipc.TestBlockingIPC#testAsyncTimeout Thread=16, OpenFileDescriptor=294, MaxFileDescriptor=60000, SystemLoadAverage=194, ProcessCount=168, AvailableMemoryMB=2868 2024-02-07 10:55:20,312 INFO [Time-limited test] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.SimpleRpcServer hosting hbase.test.pb.TestProtobufRpcProto 2024-02-07 10:55:20,313 INFO [Time-limited test] ipc.SimpleRpcServer$Listener(134): testRpcServer: started 3 reader(s) listening on port=42705 2024-02-07 10:55:20,313 DEBUG [Time-limited test] ipc.AbstractRpcClient(189): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@578e656, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2024-02-07 10:55:20,314 INFO [Time-limited test] ipc.FifoRpcScheduler(61): Using FifoRpcScheduler as user call queue; handlerCount=1; maxQueueLength=10 2024-02-07 10:55:20,314 INFO [Listener,port=42705] ipc.SimpleRpcServer$Listener(214): Listener,port=42705: starting 2024-02-07 10:55:20,314 DEBUG [RpcServer.responder] ipc.SimpleRpcServerResponder(58): RpcServer.responder: starting 2024-02-07 10:55:20,315 DEBUG [Time-limited test] ipc.RpcConnection(122): Using SIMPLE authentication for service=TestProtobufRpcProto, sasl=false 2024-02-07 10:55:20,315 DEBUG [Time-limited test] ipc.BlockingRpcConnection(501): Connecting to localhost:42705 2024-02-07 10:55:20,316 INFO [Reader=1,bindAddress=localhost,port=42705] ipc.ServerRpcConnection(540): Connection from 127.0.0.1:42920, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=TestProtobufRpcProto 2024-02-07 10:55:20,421 INFO [Time-limited test] ipc.AbstractTestIPC(411): Caught expected exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost:42705 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=0,methodName=pause], waitTime=105ms, rpcTimeout=100ms 2024-02-07 10:55:20,421 INFO [Time-limited test] ipc.AbstractTestIPC(411): Caught expected exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost:42705 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=1,methodName=pause], waitTime=104ms, rpcTimeout=100ms 2024-02-07 10:55:20,421 INFO [Time-limited test] ipc.AbstractTestIPC(411): Caught expected exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost:42705 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=2,methodName=pause], waitTime=104ms, rpcTimeout=100ms 2024-02-07 10:55:20,421 INFO [Time-limited test] ipc.AbstractTestIPC(411): Caught expected exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost:42705 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=3,methodName=pause], waitTime=104ms, rpcTimeout=100ms 2024-02-07 10:55:20,421 INFO [Time-limited test] ipc.AbstractTestIPC(411): Caught expected exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost:42705 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=4,methodName=pause], waitTime=103ms, rpcTimeout=100ms 2024-02-07 10:55:20,421 INFO [Time-limited test] ipc.AbstractTestIPC(411): Caught expected exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost:42705 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=5,methodName=pause], waitTime=103ms, rpcTimeout=100ms 2024-02-07 10:55:20,422 INFO [Time-limited test] ipc.AbstractTestIPC(411): Caught expected exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost:42705 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=6,methodName=pause], waitTime=103ms, rpcTimeout=100ms 2024-02-07 10:55:20,422 INFO [Time-limited test] ipc.AbstractTestIPC(411): Caught expected exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost:42705 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=7,methodName=pause], waitTime=103ms, rpcTimeout=100ms 2024-02-07 10:55:20,422 INFO [Time-limited test] ipc.AbstractTestIPC(411): Caught expected exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost:42705 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=8,methodName=pause], waitTime=103ms, rpcTimeout=100ms 2024-02-07 10:55:20,422 INFO [Time-limited test] ipc.AbstractTestIPC(411): Caught expected exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=localhost:42705 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=9,methodName=pause], waitTime=104ms, rpcTimeout=100ms 2024-02-07 10:55:20,422 DEBUG [Time-limited test] ipc.AbstractRpcClient(489): Stopping rpc client 2024-02-07 10:55:20,422 INFO [Time-limited test] ipc.SimpleRpcServer(438): Stopping server on 42705 2024-02-07 10:55:20,423 INFO [Listener,port=42705] ipc.SimpleRpcServer$Listener(258): Listener,port=42705: stopping 2024-02-07 10:55:20,423 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(158): RpcServer.responder: stopped 2024-02-07 10:55:20,424 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(62): RpcServer.responder: stopping 2024-02-07 10:55:20,431 INFO [Time-limited test] hbase.ResourceChecker(175): after: ipc.TestBlockingIPC#testAsyncTimeout Thread=18 (was 16) Potentially hanging thread: RpcServer.responder java.lang.Throwable.fillInStackTrace(Native Method) java.lang.Throwable.fillInStackTrace(Throwable.java:784) java.lang.Throwable.(Throwable.java:251) org.apache.log4j.spi.LoggingEvent.getLocationInformation(LoggingEvent.java:260) org.apache.log4j.helpers.PatternParser$ClassNamePatternConverter.getFullyQualifiedName(PatternParser.java:518) org.apache.log4j.helpers.PatternParser$NamedPatternConverter.convert(PatternParser.java:491) org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:63) org.apache.log4j.PatternLayout.format(PatternLayout.java:510) org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:303) org.apache.log4j.WriterAppender.append(WriterAppender.java:156) org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:232) org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:57) org.apache.log4j.Category.callAppenders(Category.java:206) org.apache.log4j.Category.forcedLog(Category.java:384) org.apache.log4j.Category.log(Category.java:810) org.slf4j.impl.Reload4jLoggerAdapter.info(Reload4jLoggerAdapter.java:284) org.apache.hadoop.hbase.ipc.SimpleRpcServerResponder.run(SimpleRpcServerResponder.java:62) - Thread LEAK? -, OpenFileDescriptor=294 (was 294), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=194 (was 194), ProcessCount=168 (was 168), AvailableMemoryMB=2849 (was 2868)