Simulation seed: 1716946672274L Forcing maxSeen: -1, maxComplete: -1, reorderBuffer: null ERROR [isolatedExecutor:2] node1 2024-05-29 01:38:14,725 SystemInfo.java:121 - 'Max processes' not found in /proc/343/limits ERROR [isolatedExecutor:2] node2 2024-05-29 01:38:23,131 SystemInfo.java:121 - 'Max processes' not found in /proc/343/limits ERROR [isolatedExecutor:2] node3 2024-05-29 01:38:28,318 SystemInfo.java:121 - 'Max processes' not found in /proc/343/limits ERROR [isolatedExecutor:2] node4 2024-05-29 01:38:32,949 SystemInfo.java:121 - 'Max processes' not found in /proc/343/limits ERROR [isolatedExecutor:2] node5 2024-05-29 01:38:36,934 SystemInfo.java:121 - 'Max processes' not found in /proc/343/limits ERROR [isolatedExecutor:2] node6 2024-05-29 01:38:40,843 SystemInfo.java:121 - 'Max processes' not found in /proc/343/limits ERROR [isolatedExecutor:2] node7 2024-05-29 01:38:44,788 SystemInfo.java:121 - 'Max processes' not found in /proc/343/limits ERROR [isolatedExecutor:2] node8 2024-05-29 01:38:48,967 SystemInfo.java:121 - 'Max processes' not found in /proc/343/limits ERROR [isolatedExecutor:2] node9 2024-05-29 01:38:53,097 SystemInfo.java:121 - 'Max processes' not found in /proc/343/limits ERROR [isolatedExecutor:2] node10 2024-05-29 01:38:57,405 SystemInfo.java:121 - 'Max processes' not found in /proc/343/limits ERROR [isolatedExecutor:2] node11 2024-05-29 01:39:01,984 SystemInfo.java:121 - 'Max processes' not found in /proc/343/limits ERROR [isolatedExecutor:2] node12 2024-05-29 01:39:05,849 SystemInfo.java:121 - 'Max processes' not found in /proc/343/limits WARN [OptionalTasks:1] node1 2024-05-29 01:39:09,580 SystemKeyspace.java:1195 - Using stored Gossip Generation 1577836803 as it is greater than current system time 1577836802. See CASSANDRA-3654 if you experience problems WARN [OptionalTasks:1] node2 2024-05-29 01:39:11,047 SystemKeyspace.java:1195 - Using stored Gossip Generation 1577836803 as it is greater than current system time 1577836802. See CASSANDRA-3654 if you experience problems WARN [OptionalTasks:1] node3 2024-05-29 01:39:11,846 SystemKeyspace.java:1195 - Using stored Gossip Generation 1577836803 as it is greater than current system time 1577836802. See CASSANDRA-3654 if you experience problems WARN [OptionalTasks:1] node5 2024-05-29 01:39:12,503 SystemKeyspace.java:1195 - Using stored Gossip Generation 1577836801 as it is greater than current system time 1577836801. See CASSANDRA-3654 if you experience problems WARN [OptionalTasks:1] node5 2024-05-29 01:39:12,534 SystemKeyspace.java:1195 - Using stored Gossip Generation 1577836802 as it is greater than current system time 1577836801. See CASSANDRA-3654 if you experience problems WARN [OptionalTasks:1] node6 2024-05-29 01:39:13,279 SystemKeyspace.java:1195 - Using stored Gossip Generation 1577836801 as it is greater than current system time 1577836801. See CASSANDRA-3654 if you experience problems WARN [OptionalTasks:1] node6 2024-05-29 01:39:13,307 SystemKeyspace.java:1195 - Using stored Gossip Generation 1577836802 as it is greater than current system time 1577836801. See CASSANDRA-3654 if you experience problems WARN [OptionalTasks:1] node7 2024-05-29 01:39:14,023 SystemKeyspace.java:1195 - Using stored Gossip Generation 1577836801 as it is greater than current system time 1577836801. See CASSANDRA-3654 if you experience problems WARN [OptionalTasks:1] node7 2024-05-29 01:39:14,055 SystemKeyspace.java:1195 - Using stored Gossip Generation 1577836802 as it is greater than current system time 1577836801. See CASSANDRA-3654 if you experience problems WARN [OptionalTasks:1] node9 2024-05-29 01:39:14,848 SystemKeyspace.java:1195 - Using stored Gossip Generation 1577836801 as it is greater than current system time 1577836801. See CASSANDRA-3654 if you experience problems WARN [OptionalTasks:1] node9 2024-05-29 01:39:14,888 SystemKeyspace.java:1195 - Using stored Gossip Generation 1577836802 as it is greater than current system time 1577836801. See CASSANDRA-3654 if you experience problems WARN [OptionalTasks:1] node10 2024-05-29 01:39:15,763 SystemKeyspace.java:1195 - Using stored Gossip Generation 1577836801 as it is greater than current system time 1577836801. See CASSANDRA-3654 if you experience problems WARN [OptionalTasks:1] node10 2024-05-29 01:39:15,807 SystemKeyspace.java:1195 - Using stored Gossip Generation 1577836802 as it is greater than current system time 1577836801. See CASSANDRA-3654 if you experience problems WARN [OptionalTasks:1] node11 2024-05-29 01:39:16,873 SystemKeyspace.java:1195 - Using stored Gossip Generation 1577836802 as it is greater than current system time 1577836801. See CASSANDRA-3654 if you experience problems WARN [OptionalTasks:1] node11 2024-05-29 01:39:16,911 SystemKeyspace.java:1195 - Using stored Gossip Generation 1577836803 as it is greater than current system time 1577836801. See CASSANDRA-3654 if you experience problems WARN [isolatedExecutor:1] node1 2024-05-29 01:39:32,508 HarrySimulatorTest.java:287 - Epoch{epoch=37} WARN [isolatedExecutor:2] node2 2024-05-29 01:39:32,509 HarrySimulatorTest.java:287 - Epoch{epoch=37} WARN [isolatedExecutor:2] node3 2024-05-29 01:39:32,510 HarrySimulatorTest.java:287 - Epoch{epoch=37} WARN [isolatedExecutor:1] node4 2024-05-29 01:39:32,510 HarrySimulatorTest.java:287 - Epoch{epoch=37} WARN [isolatedExecutor:2] node5 2024-05-29 01:39:32,511 HarrySimulatorTest.java:287 - Epoch{epoch=37} WARN [isolatedExecutor:2] node6 2024-05-29 01:39:32,512 HarrySimulatorTest.java:287 - Epoch{epoch=37} WARN [isolatedExecutor:2] node7 2024-05-29 01:39:32,512 HarrySimulatorTest.java:287 - Epoch{epoch=37} WARN [isolatedExecutor:1] node8 2024-05-29 01:39:32,513 HarrySimulatorTest.java:287 - Epoch{epoch=37} WARN [isolatedExecutor:2] node9 2024-05-29 01:39:32,518 HarrySimulatorTest.java:287 - Epoch{epoch=37} WARN [isolatedExecutor:2] node10 2024-05-29 01:39:32,518 HarrySimulatorTest.java:287 - Epoch{epoch=37} WARN [isolatedExecutor:2] node11 2024-05-29 01:39:32,519 HarrySimulatorTest.java:287 - Epoch{epoch=37} WARN [isolatedExecutor:1] node12 2024-05-29 01:39:32,519 HarrySimulatorTest.java:287 - Epoch{epoch=37} WARN [main] <main> 2024-05-29 01:39:32,658 HarrySimulatorTest.java:781 - Starting validation of 0 written partitions. Highest LTS is 0. Ring view: SimulatedNodeState{ring=[dc0-/127.0.0.1:7012@-7686143364045646507, dc0-/127.0.0.2:7012@-6148914691236517207, dc0-/127.0.0.3:7012@-4611686018427387907, dc1-/127.0.0.5:7012@-1537228672809129307, dc1-/127.0.0.6:7012@-7, dc1-/127.0.0.7:7012@1537228672809129293, dc2-/127.0.0.9:7012@4611686018427387893, dc2-/127.0.0.10:7012@6148914691236517193, dc2-/127.0.0.11:7012@7686143364045646493]} WARN [MetadataFetchLogStage:1] node1 2024-05-29 01:39:34,597 RemoteProcessor.java:238 - Got error from /127.0.0.5:7012: TIMEOUT when sending TCM_FETCH_PEER_LOG_REQ, retrying on CandidateIterator{candidates=[/127.0.0.5:7012], checkLive=false} WARN [MetadataFetchLogStage:1] node6 2024-05-29 01:39:34,746 RemoteProcessor.java:238 - Got error from /127.0.0.5:7012: TIMEOUT when sending TCM_FETCH_PEER_LOG_REQ, retrying on CandidateIterator{candidates=[/127.0.0.5:7012], checkLive=false} WARN [OptionalTasks:1] node4 2024-05-29 01:39:36,704 SystemKeyspace.java:1195 - Using stored Gossip Generation 1577851237 as it is greater than current system time 1577851236. See CASSANDRA-3654 if you experience problems WARN [isolatedExecutor:1] node1 2024-05-29 01:39:36,812 HarrySimulatorTest.java:319 - Epoch{epoch=41} WARN [isolatedExecutor:2] node2 2024-05-29 01:39:36,812 HarrySimulatorTest.java:319 - Epoch{epoch=41} WARN [isolatedExecutor:1] node3 2024-05-29 01:39:36,812 HarrySimulatorTest.java:319 - Epoch{epoch=41} WARN [isolatedExecutor:1] node4 2024-05-29 01:39:36,812 HarrySimulatorTest.java:319 - Epoch{epoch=41} WARN [isolatedExecutor:1] node5 2024-05-29 01:39:36,813 HarrySimulatorTest.java:319 - Epoch{epoch=41} WARN [isolatedExecutor:1] node6 2024-05-29 01:39:36,813 HarrySimulatorTest.java:319 - Epoch{epoch=41} WARN [isolatedExecutor:1] node7 2024-05-29 01:39:36,813 HarrySimulatorTest.java:319 - Epoch{epoch=41} WARN [isolatedExecutor:2] node8 2024-05-29 01:39:36,813 HarrySimulatorTest.java:319 - Epoch{epoch=41} WARN [isolatedExecutor:1] node9 2024-05-29 01:39:36,816 HarrySimulatorTest.java:319 - Epoch{epoch=41} WARN [isolatedExecutor:1] node10 2024-05-29 01:39:36,818 HarrySimulatorTest.java:319 - Epoch{epoch=41} WARN [isolatedExecutor:1] node11 2024-05-29 01:39:36,819 HarrySimulatorTest.java:319 - Epoch{epoch=41} WARN [isolatedExecutor:2] node12 2024-05-29 01:39:36,819 HarrySimulatorTest.java:319 - Epoch{epoch=41} Marking 4 as bootstrapped WARN [main] <main> 2024-05-29 01:39:36,820 HarrySimulatorTest.java:781 - Starting validation of 1 written partitions. Highest LTS is 1. Ring view: SimulatedNodeState{ring=[dc0-/127.0.0.1:7012@-7686143364045646507, dc0-/127.0.0.2:7012@-6148914691236517207, dc0-/127.0.0.3:7012@-4611686018427387907, dc1-/127.0.0.5:7012@-1537228672809129307, dc1-/127.0.0.6:7012@-7, dc1-/127.0.0.7:7012@1537228672809129293, dc2-/127.0.0.9:7012@4611686018427387893, dc2-/127.0.0.10:7012@6148914691236517193, dc0-/127.0.0.4:7012@6457576751661366678, dc2-/127.0.0.11:7012@7686143364045646493]} WARN [main] <main> 2024-05-29 01:39:36,945 HarrySimulatorTest.java:347 - Finished bootstrap of 4 and data validation! WARN [InternalMetadataStage:2] node1 2024-05-29 01:39:37,223 DistributedMetadataLogKeyspace.java:146 - Timed out while trying to append item to the log org.apache.cassandra.exceptions.CasWriteTimeoutException: CAS operation timed out: received 3 of 4 required responses after 0 contention retries at org.apache.cassandra.service.paxos.Paxos$MaybeFailure.markAndThrowAsTimeoutOrFailure(Paxos.java:613) at org.apache.cassandra.service.paxos.Paxos.begin(Paxos.java:1118) at org.apache.cassandra.service.paxos.Paxos.cas(Paxos.java:725) at org.apache.cassandra.service.paxos.Paxos.cas(Paxos.java:684) at org.apache.cassandra.service.StorageProxy.cas(StorageProxy.java:322) at org.apache.cassandra.cql3.statements.ModificationStatement.executeWithCondition(ModificationStatement.java:542) at org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:501) at org.apache.cassandra.cql3.QueryProcessor.execute(QueryProcessor.java:561) at org.apache.cassandra.cql3.QueryProcessor.execute(QueryProcessor.java:542) at org.apache.cassandra.schema.DistributedMetadataLogKeyspace.tryCommit(DistributedMetadataLogKeyspace.java:135) at org.apache.cassandra.tcm.PaxosBackedProcessor.tryCommitOne(PaxosBackedProcessor.java:69) at org.apache.cassandra.tcm.AbstractLocalProcessor.commit(AbstractLocalProcessor.java:93) at org.apache.cassandra.distributed.test.log.TestProcessor.commit(TestProcessor.java:61) at org.apache.cassandra.tcm.Commit$Handler.doVerb(Commit.java:368) at org.apache.cassandra.net.InboundSink.lambda$new$0(InboundSink.java:102) at org.apache.cassandra.net.InboundSink$Filtered.accept(InboundSink.java:70) at org.apache.cassandra.net.InboundSink$Filtered.accept(InboundSink.java:56) at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:122) at org.apache.cassandra.distributed.impl.Instance.lambda$receiveMessageRunnable$43231af8$1(Instance.java:543) at org.apache.cassandra.distributed.impl.Instance.receiveMessageWithInvokingThread(Instance.java:513) at org.apache.cassandra.distributed.impl.AbstractCluster$Wrapper.receiveMessageWithInvokingThread(AbstractCluster.java:497) at org.apache.cassandra.simulator.systems.SimulatedAction.lambda$applyToMessage$3(SimulatedAction.java:355) at org.apache.cassandra.simulator.systems.InterceptedExecution$InterceptedRunnableExecution.run(InterceptedExecution.java:118) at org.apache.cassandra.simulator.systems.InterceptingExecutor$InterceptingPooledExecutor$WaitingThread.lambda$new$1(InterceptingExecutor.java:318) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) WARN [InternalMetadataStage:1] node8 2024-05-29 01:39:39,207 RemoteProcessor.java:238 - Got error from /127.0.0.1:7012: TIMEOUT when sending TCM_COMMIT_REQ, retrying on CandidateIterator{candidates=[/127.0.0.2:7012, /127.0.0.6:7012, /127.0.0.5:7012, /127.0.0.9:7012, /127.0.0.10:7012, /127.0.0.1:7012], checkLive=true} WARN [InternalMetadataStage:1] node8 2024-05-29 01:39:39,912 RemoteProcessor.java:238 - Got error from /127.0.0.2:7012: TIMEOUT when sending TCM_COMMIT_REQ, retrying on CandidateIterator{candidates=[/127.0.0.6:7012, /127.0.0.9:7012, /127.0.0.10:7012, /127.0.0.1:7012, /127.0.0.5:7012, /127.0.0.2:7012], checkLive=true} ERROR [MutationStage:1] <main> 2024-05-29 01:39:42,004 HarrySimulatorTest.java:980 - Caught an exception, going to halt java.lang.OutOfMemoryError: Direct buffer memory at java.base/java.nio.Bits.reserveMemory(Bits.java:175) at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) at org.apache.cassandra.hints.HintsBuffer.create(HintsBuffer.java:77) at org.apache.cassandra.hints.HintsBufferPool.createBuffer(HintsBufferPool.java:133) at org.apache.cassandra.hints.HintsBufferPool.initializeCurrentBuffer(HintsBufferPool.java:104) at org.apache.cassandra.hints.HintsBufferPool.currentBuffer(HintsBufferPool.java:96) at org.apache.cassandra.hints.HintsBufferPool.allocate(HintsBufferPool.java:70) at org.apache.cassandra.hints.HintsBufferPool.write(HintsBufferPool.java:62) at org.apache.cassandra.hints.HintsService.write(HintsService.java:169) at org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:2724) at org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:2641) at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96) at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61) at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71) at org.apache.cassandra.simulator.systems.InterceptingExecutor$InterceptingPooledExecutor$WaitingThread.lambda$new$1(InterceptingExecutor.java:318) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) WARN [OptionalTasks:1] node8 2024-05-29 01:39:42,587 SystemKeyspace.java:1195 - Using stored Gossip Generation 1577854859 as it is greater than current system time 1577854858. See CASSANDRA-3654 if you experience problems WARN [isolatedExecutor:1] node1 2024-05-29 01:39:42,701 HarrySimulatorTest.java:319 - Epoch{epoch=45} WARN [isolatedExecutor:1] node2 2024-05-29 01:39:42,702 HarrySimulatorTest.java:319 - Epoch{epoch=45} WARN [isolatedExecutor:1] node3 2024-05-29 01:39:42,702 HarrySimulatorTest.java:319 - Epoch{epoch=45} WARN [isolatedExecutor:1] node4 2024-05-29 01:39:42,702 HarrySimulatorTest.java:319 - Epoch{epoch=45} WARN [isolatedExecutor:1] node5 2024-05-29 01:39:42,702 HarrySimulatorTest.java:319 - Epoch{epoch=45} WARN [isolatedExecutor:2] node6 2024-05-29 01:39:42,703 HarrySimulatorTest.java:319 - Epoch{epoch=45} WARN [isolatedExecutor:2] node7 2024-05-29 01:39:42,703 HarrySimulatorTest.java:319 - Epoch{epoch=45} WARN [isolatedExecutor:2] node8 2024-05-29 01:39:42,703 HarrySimulatorTest.java:319 - Epoch{epoch=45} WARN [isolatedExecutor:1] node9 2024-05-29 01:39:42,703 HarrySimulatorTest.java:319 - Epoch{epoch=45} WARN [isolatedExecutor:2] node10 2024-05-29 01:39:42,704 HarrySimulatorTest.java:319 - Epoch{epoch=45} WARN [isolatedExecutor:2] node11 2024-05-29 01:39:42,704 HarrySimulatorTest.java:319 - Epoch{epoch=44} WARN [isolatedExecutor:1] node12 2024-05-29 01:39:42,704 HarrySimulatorTest.java:319 - Epoch{epoch=45} Marking 8 as bootstrapped WARN [main] <main> 2024-05-29 01:39:42,704 HarrySimulatorTest.java:781 - Starting validation of 2 written partitions. Highest LTS is 2. Ring view: SimulatedNodeState{ring=[dc0-/127.0.0.1:7012@-7686143364045646507, dc0-/127.0.0.2:7012@-6148914691236517207, dc0-/127.0.0.3:7012@-4611686018427387907, dc1-/127.0.0.5:7012@-1537228672809129307, dc1-/127.0.0.6:7012@-7, dc1-/127.0.0.7:7012@1537228672809129293, dc1-/127.0.0.8:7012@3507966432223637333, dc2-/127.0.0.9:7012@4611686018427387893, dc2-/127.0.0.10:7012@6148914691236517193, dc0-/127.0.0.4:7012@6457576751661366678, dc2-/127.0.0.11:7012@7686143364045646493]} WARN [main] <main> 2024-05-29 01:39:42,721 HarrySimulatorTest.java:347 - Finished bootstrap of 8 and data validation! ERROR [MutationStage:1] <main> 2024-05-29 01:39:45,712 HarrySimulatorTest.java:980 - Caught an exception, going to halt java.lang.OutOfMemoryError: Direct buffer memory at java.base/java.nio.Bits.reserveMemory(Bits.java:175) at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) at org.apache.cassandra.hints.HintsBuffer.create(HintsBuffer.java:77) at org.apache.cassandra.hints.HintsBufferPool.createBuffer(HintsBufferPool.java:133) at org.apache.cassandra.hints.HintsBufferPool.initializeCurrentBuffer(HintsBufferPool.java:104) at org.apache.cassandra.hints.HintsBufferPool.currentBuffer(HintsBufferPool.java:96) at org.apache.cassandra.hints.HintsBufferPool.allocate(HintsBufferPool.java:70) at org.apache.cassandra.hints.HintsBufferPool.write(HintsBufferPool.java:62) at org.apache.cassandra.hints.HintsService.write(HintsService.java:169) at org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:2724) at org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:2641) at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96) at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61) at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71) at org.apache.cassandra.simulator.systems.InterceptingExecutor$InterceptingPooledExecutor$WaitingThread.lambda$new$1(InterceptingExecutor.java:318) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) ERROR [HintsWriteExecutor:1] <main> 2024-05-29 01:39:47,036 HarrySimulatorTest.java:980 - Caught an exception, going to halt java.lang.OutOfMemoryError: Direct buffer memory at java.base/java.nio.Bits.reserveMemory(Bits.java:175) at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) at org.apache.cassandra.hints.HintsBuffer.create(HintsBuffer.java:77) at org.apache.cassandra.hints.HintsBufferPool.createBuffer(HintsBufferPool.java:133) at org.apache.cassandra.hints.HintsBufferPool.initializeCurrentBuffer(HintsBufferPool.java:104) at org.apache.cassandra.hints.HintsBufferPool.currentBuffer(HintsBufferPool.java:96) at org.apache.cassandra.hints.HintsWriteExecutor$FlushBufferPoolTask.run(HintsWriteExecutor.java:168) at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96) at org.apache.cassandra.concurrent.SyncFutureTask.run(SyncFutureTask.java:68) at org.apache.cassandra.simulator.systems.InterceptingExecutor$AbstractSingleThreadedExecutorPlus.lambda$new$0(InterceptingExecutor.java:585) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) WARN [InternalMetadataStage:1] node12 2024-05-29 01:39:47,602 RemoteProcessor.java:238 - Got error from /127.0.0.1:7012: TIMEOUT when sending TCM_COMMIT_REQ, retrying on CandidateIterator{candidates=[/127.0.0.6:7012, /127.0.0.2:7012, /127.0.0.9:7012, /127.0.0.5:7012, /127.0.0.10:7012, /127.0.0.1:7012], checkLive=true} WARN [MetadataFetchLogStage:1] node1 2024-05-29 01:39:48,073 RemoteProcessor.java:238 - Got error from /127.0.0.12:7012: TIMEOUT when sending TCM_FETCH_PEER_LOG_REQ, retrying on CandidateIterator{candidates=[/127.0.0.12:7012], checkLive=false} WARN [OptionalTasks:1] node12 2024-05-29 01:39:48,449 SystemKeyspace.java:1195 - Using stored Gossip Generation 1577858460 as it is greater than current system time 1577858459. See CASSANDRA-3654 if you experience problems WARN [isolatedExecutor:2] node1 2024-05-29 01:39:48,653 HarrySimulatorTest.java:319 - Epoch{epoch=49} WARN [isolatedExecutor:2] node2 2024-05-29 01:39:48,653 HarrySimulatorTest.java:319 - Epoch{epoch=49} WARN [isolatedExecutor:2] node3 2024-05-29 01:39:48,653 HarrySimulatorTest.java:319 - Epoch{epoch=49} WARN [isolatedExecutor:1] node4 2024-05-29 01:39:48,653 HarrySimulatorTest.java:319 - Epoch{epoch=49} WARN [isolatedExecutor:2] node5 2024-05-29 01:39:48,653 HarrySimulatorTest.java:319 - Epoch{epoch=49} WARN [isolatedExecutor:1] node6 2024-05-29 01:39:48,654 HarrySimulatorTest.java:319 - Epoch{epoch=49} WARN [isolatedExecutor:1] node7 2024-05-29 01:39:48,654 HarrySimulatorTest.java:319 - Epoch{epoch=48} WARN [isolatedExecutor:2] node8 2024-05-29 01:39:48,654 HarrySimulatorTest.java:319 - Epoch{epoch=48} WARN [isolatedExecutor:2] node9 2024-05-29 01:39:48,654 HarrySimulatorTest.java:319 - Epoch{epoch=49} WARN [isolatedExecutor:1] node10 2024-05-29 01:39:48,655 HarrySimulatorTest.java:319 - Epoch{epoch=49} WARN [isolatedExecutor:1] node11 2024-05-29 01:39:48,655 HarrySimulatorTest.java:319 - Epoch{epoch=49} WARN [isolatedExecutor:1] node12 2024-05-29 01:39:48,655 HarrySimulatorTest.java:319 - Epoch{epoch=49} Marking 12 as bootstrapped WARN [main] <main> 2024-05-29 01:39:48,655 HarrySimulatorTest.java:781 - Starting validation of 3 written partitions. Highest LTS is 3. Ring view: SimulatedNodeState{ring=[dc0-/127.0.0.1:7012@-7686143364045646507, dc0-/127.0.0.2:7012@-6148914691236517207, dc0-/127.0.0.3:7012@-4611686018427387907, dc1-/127.0.0.5:7012@-1537228672809129307, dc1-/127.0.0.6:7012@-7, dc1-/127.0.0.7:7012@1537228672809129293, dc1-/127.0.0.8:7012@3507966432223637333, dc2-/127.0.0.12:7012@3890885889187058965, dc2-/127.0.0.9:7012@4611686018427387893, dc2-/127.0.0.10:7012@6148914691236517193, dc0-/127.0.0.4:7012@6457576751661366678, dc2-/127.0.0.11:7012@7686143364045646493]} WARN [main] <main> 2024-05-29 01:39:48,738 HarrySimulatorTest.java:347 - Finished bootstrap of 12 and data validation! WARN [MetadataFetchLogStage:1] node8 2024-05-29 01:39:49,607 RemoteProcessor.java:238 - Got error from /127.0.0.4:7012: TIMEOUT when sending TCM_FETCH_PEER_LOG_REQ, retrying on CandidateIterator{candidates=[/127.0.0.4:7012], checkLive=false} ERROR [HintsWriteExecutor:1] <main> 2024-05-29 01:39:51,192 HarrySimulatorTest.java:980 - Caught an exception, going to halt java.lang.OutOfMemoryError: Direct buffer memory at java.base/java.nio.Bits.reserveMemory(Bits.java:175) at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) at org.apache.cassandra.hints.HintsBuffer.create(HintsBuffer.java:77) at org.apache.cassandra.hints.HintsBufferPool.createBuffer(HintsBufferPool.java:133) at org.apache.cassandra.hints.HintsBufferPool.initializeCurrentBuffer(HintsBufferPool.java:104) at org.apache.cassandra.hints.HintsBufferPool.currentBuffer(HintsBufferPool.java:96) at org.apache.cassandra.hints.HintsWriteExecutor$PartiallyFlushBufferPoolTask.run(HintsWriteExecutor.java:195) at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96) at org.apache.cassandra.concurrent.SyncFutureTask.run(SyncFutureTask.java:68) at org.apache.cassandra.simulator.systems.InterceptingExecutor$AbstractSingleThreadedExecutorPlus.lambda$new$0(InterceptingExecutor.java:585) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) ERROR [HintsWriteExecutor:1] <main> 2024-05-29 01:39:52,522 HarrySimulatorTest.java:980 - Caught an exception, going to halt java.lang.OutOfMemoryError: Direct buffer memory at java.base/java.nio.Bits.reserveMemory(Bits.java:175) at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) at org.apache.cassandra.hints.HintsBuffer.create(HintsBuffer.java:77) at org.apache.cassandra.hints.HintsBufferPool.createBuffer(HintsBufferPool.java:133) at org.apache.cassandra.hints.HintsBufferPool.initializeCurrentBuffer(HintsBufferPool.java:104) at org.apache.cassandra.hints.HintsBufferPool.currentBuffer(HintsBufferPool.java:96) at org.apache.cassandra.hints.HintsWriteExecutor$FlushBufferPoolTask.run(HintsWriteExecutor.java:168) at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96) at org.apache.cassandra.concurrent.SyncFutureTask.run(SyncFutureTask.java:68) at org.apache.cassandra.simulator.systems.InterceptingExecutor$AbstractSingleThreadedExecutorPlus.lambda$new$0(InterceptingExecutor.java:585) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) ERROR [MutationStage:1] <main> 2024-05-29 01:39:54,281 HarrySimulatorTest.java:980 - Caught an exception, going to halt java.lang.OutOfMemoryError: Direct buffer memory at java.base/java.nio.Bits.reserveMemory(Bits.java:175) at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) at org.apache.cassandra.hints.HintsBuffer.create(HintsBuffer.java:77) at org.apache.cassandra.hints.HintsBufferPool.createBuffer(HintsBufferPool.java:133) at org.apache.cassandra.hints.HintsBufferPool.initializeCurrentBuffer(HintsBufferPool.java:104) at org.apache.cassandra.hints.HintsBufferPool.currentBuffer(HintsBufferPool.java:96) at org.apache.cassandra.hints.HintsBufferPool.allocate(HintsBufferPool.java:70) at org.apache.cassandra.hints.HintsBufferPool.write(HintsBufferPool.java:62) at org.apache.cassandra.hints.HintsService.write(HintsService.java:169) at org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:2724) at org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:2641) at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96) at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61) at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71) at org.apache.cassandra.simulator.systems.InterceptingExecutor$InterceptingPooledExecutor$WaitingThread.lambda$new$1(InterceptingExecutor.java:318) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) ERROR [MutationStage:1] <main> 2024-05-29 01:39:55,515 HarrySimulatorTest.java:980 - Caught an exception, going to halt java.lang.OutOfMemoryError: Direct buffer memory at java.base/java.nio.Bits.reserveMemory(Bits.java:175) at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) at org.apache.cassandra.hints.HintsBuffer.create(HintsBuffer.java:77) at org.apache.cassandra.hints.HintsBufferPool.createBuffer(HintsBufferPool.java:133) at org.apache.cassandra.hints.HintsBufferPool.initializeCurrentBuffer(HintsBufferPool.java:104) at org.apache.cassandra.hints.HintsBufferPool.currentBuffer(HintsBufferPool.java:96) at org.apache.cassandra.hints.HintsBufferPool.allocate(HintsBufferPool.java:70) at org.apache.cassandra.hints.HintsBufferPool.write(HintsBufferPool.java:62) at org.apache.cassandra.hints.HintsService.write(HintsService.java:169) at org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:2724) at org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:2641) at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96) at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61) at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71) at org.apache.cassandra.simulator.systems.InterceptingExecutor$InterceptingPooledExecutor$WaitingThread.lambda$new$1(InterceptingExecutor.java:318) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) ERROR [HintsWriteExecutor:1] <main> 2024-05-29 01:39:56,763 HarrySimulatorTest.java:980 - Caught an exception, going to halt java.lang.OutOfMemoryError: Direct buffer memory at java.base/java.nio.Bits.reserveMemory(Bits.java:175) at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) at org.apache.cassandra.hints.HintsBuffer.create(HintsBuffer.java:77) at org.apache.cassandra.hints.HintsBufferPool.createBuffer(HintsBufferPool.java:133) at org.apache.cassandra.hints.HintsBufferPool.initializeCurrentBuffer(HintsBufferPool.java:104) at org.apache.cassandra.hints.HintsBufferPool.currentBuffer(HintsBufferPool.java:96) at org.apache.cassandra.hints.HintsWriteExecutor$PartiallyFlushBufferPoolTask.run(HintsWriteExecutor.java:195) at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96) at org.apache.cassandra.concurrent.SyncFutureTask.run(SyncFutureTask.java:68) at org.apache.cassandra.simulator.systems.InterceptingExecutor$AbstractSingleThreadedExecutorPlus.lambda$new$0(InterceptingExecutor.java:585) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) ERROR [HintsWriteExecutor:1] <main> 2024-05-29 01:39:58,023 HarrySimulatorTest.java:980 - Caught an exception, going to halt java.lang.OutOfMemoryError: Direct buffer memory at java.base/java.nio.Bits.reserveMemory(Bits.java:175) at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) at org.apache.cassandra.hints.HintsBuffer.create(HintsBuffer.java:77) at org.apache.cassandra.hints.HintsBufferPool.createBuffer(HintsBufferPool.java:133) at org.apache.cassandra.hints.HintsBufferPool.initializeCurrentBuffer(HintsBufferPool.java:104) at org.apache.cassandra.hints.HintsBufferPool.currentBuffer(HintsBufferPool.java:96) at org.apache.cassandra.hints.HintsWriteExecutor$FlushBufferPoolTask.run(HintsWriteExecutor.java:168) at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96) at org.apache.cassandra.concurrent.SyncFutureTask.run(SyncFutureTask.java:68) at org.apache.cassandra.simulator.systems.InterceptingExecutor$AbstractSingleThreadedExecutorPlus.lambda$new$0(InterceptingExecutor.java:585) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) WARN [isolatedExecutor:2] node4 2024-05-29 01:39:58,028 UnbootstrapAndLeave.java:193 - Error committing midLeave java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Direct buffer memory at org.apache.cassandra.hints.HintsService.transferHints(HintsService.java:435) at org.apache.cassandra.service.StorageService.streamHints(StorageService.java:3597) at org.apache.cassandra.tcm.sequences.UnbootstrapStreams.unbootstrap(UnbootstrapStreams.java:130) at org.apache.cassandra.tcm.sequences.UnbootstrapStreams.execute(UnbootstrapStreams.java:74) at org.apache.cassandra.tcm.sequences.UnbootstrapAndLeave.executeNext(UnbootstrapAndLeave.java:178) at org.apache.cassandra.tcm.sequences.InProgressSequences.resume(InProgressSequences.java:200) at org.apache.cassandra.tcm.sequences.InProgressSequences.finishInProgressSequences(InProgressSequences.java:72) at org.apache.cassandra.tcm.sequences.SingleNodeSequences.decommission(SingleNodeSequences.java:89) at org.apache.cassandra.simulator.test.HarrySimulatorTest.lambda$decommission$6ea5f628$1(HarrySimulatorTest.java:645) at org.apache.cassandra.simulator.systems.InterceptedExecution$InterceptedRunnableExecution.run(InterceptedExecution.java:118) at org.apache.cassandra.simulator.systems.InterceptingExecutor$InterceptingPooledExecutor$WaitingThread.lambda$new$1(InterceptingExecutor.java:318) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Direct buffer memory at org.apache.cassandra.utils.concurrent.AbstractFuture.getWhenDone(AbstractFuture.java:239) at org.apache.cassandra.utils.concurrent.AbstractFuture.get(AbstractFuture.java:246) at org.apache.cassandra.hints.HintsService.transferHints(HintsService.java:426) ... 12 common frames omitted Caused by: java.lang.OutOfMemoryError: Direct buffer memory at java.base/java.nio.Bits.reserveMemory(Bits.java:175) at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) at org.apache.cassandra.hints.HintsBuffer.create(HintsBuffer.java:77) at org.apache.cassandra.hints.HintsBufferPool.createBuffer(HintsBufferPool.java:133) at org.apache.cassandra.hints.HintsBufferPool.initializeCurrentBuffer(HintsBufferPool.java:104) at org.apache.cassandra.hints.HintsBufferPool.currentBuffer(HintsBufferPool.java:96) at org.apache.cassandra.hints.HintsWriteExecutor$FlushBufferPoolTask.run(HintsWriteExecutor.java:168) at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96) at org.apache.cassandra.concurrent.SyncFutureTask.run(SyncFutureTask.java:68) at org.apache.cassandra.simulator.systems.InterceptingExecutor$AbstractSingleThreadedExecutorPlus.lambda$new$0(InterceptingExecutor.java:585) ... 2 common frames omitted ERROR [isolatedExecutor:2] node4 2024-05-29 01:39:58,028 JVMStabilityInspector.java:123 - OutOfMemory error letting the JVM handle the error: java.lang.OutOfMemoryError: Direct buffer memory at java.base/java.nio.Bits.reserveMemory(Bits.java:175) at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) at org.apache.cassandra.hints.HintsBuffer.create(HintsBuffer.java:77) at org.apache.cassandra.hints.HintsBufferPool.createBuffer(HintsBufferPool.java:133) at org.apache.cassandra.hints.HintsBufferPool.initializeCurrentBuffer(HintsBufferPool.java:104) at org.apache.cassandra.hints.HintsBufferPool.currentBuffer(HintsBufferPool.java:96) at org.apache.cassandra.hints.HintsWriteExecutor$FlushBufferPoolTask.run(HintsWriteExecutor.java:168) at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96) at org.apache.cassandra.concurrent.SyncFutureTask.run(SyncFutureTask.java:68) at org.apache.cassandra.simulator.systems.InterceptingExecutor$AbstractSingleThreadedExecutorPlus.lambda$new$0(InterceptingExecutor.java:585) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) ERROR [isolatedExecutor:2] node4 2024-05-29 01:39:58,029 JVMStabilityInspector.java:186 - Force heap space OutOfMemoryError in the presence of java.lang.OutOfMemoryError: Direct buffer memory at java.base/java.nio.Bits.reserveMemory(Bits.java:175) at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) at org.apache.cassandra.hints.HintsBuffer.create(HintsBuffer.java:77) at org.apache.cassandra.hints.HintsBufferPool.createBuffer(HintsBufferPool.java:133) at org.apache.cassandra.hints.HintsBufferPool.initializeCurrentBuffer(HintsBufferPool.java:104) at org.apache.cassandra.hints.HintsBufferPool.currentBuffer(HintsBufferPool.java:96) at org.apache.cassandra.hints.HintsWriteExecutor$FlushBufferPoolTask.run(HintsWriteExecutor.java:168) at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96) at org.apache.cassandra.concurrent.SyncFutureTask.run(SyncFutureTask.java:68) at org.apache.cassandra.simulator.systems.InterceptingExecutor$AbstractSingleThreadedExecutorPlus.lambda$new$0(InterceptingExecutor.java:585) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) ERROR [isolatedExecutor:2] node4 2024-05-29 01:39:59,486 HarrySimulatorTest.java:649 - Could not decommission. Interrupting simulation. java.lang.OutOfMemoryError: Java heap space at org.apache.cassandra.utils.JVMStabilityInspector.forceHeapSpaceOomMaybe(JVMStabilityInspector.java:193) at org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:127) at org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:170) at org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:170) at org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:89) at org.apache.cassandra.tcm.sequences.UnbootstrapAndLeave.executeNext(UnbootstrapAndLeave.java:194) at org.apache.cassandra.tcm.sequences.InProgressSequences.resume(InProgressSequences.java:200) at org.apache.cassandra.tcm.sequences.InProgressSequences.finishInProgressSequences(InProgressSequences.java:72) at org.apache.cassandra.tcm.sequences.SingleNodeSequences.decommission(SingleNodeSequences.java:89) at org.apache.cassandra.simulator.test.HarrySimulatorTest.lambda$decommission$6ea5f628$1(HarrySimulatorTest.java:645) at org.apache.cassandra.simulator.test.HarrySimulatorTest$$Lambda$17807/0x00000008444bdc40.run(Unknown Source) at org.apache.cassandra.simulator.systems.InterceptedExecution$InterceptedRunnableExecution.run(InterceptedExecution.java:118) at org.apache.cassandra.simulator.systems.InterceptingExecutor$InterceptingPooledExecutor$WaitingThread.lambda$new$1(InterceptingExecutor.java:318) at org.apache.cassandra.simulator.systems.InterceptingExecutor$InterceptingPooledExecutor$WaitingThread$$Lambda$5312/0x0000000841aa2c40.run(Unknown Source) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) ERROR [isolatedExecutor:2] <main> 2024-05-29 01:39:59,489 HarrySimulatorTest.java:980 - Caught an exception, going to halt java.lang.OutOfMemoryError: Requested array size exceeds VM limit at org.apache.cassandra.simulator.SimulatorUtils.failWithOOM(SimulatorUtils.java:36) at org.apache.cassandra.simulator.test.HarrySimulatorTest.lambda$decommission$6ea5f628$1(HarrySimulatorTest.java:650) at org.apache.cassandra.simulator.test.HarrySimulatorTest$$Lambda$17807/0x00000008444bdc40.run(Unknown Source) at org.apache.cassandra.simulator.systems.InterceptedExecution$InterceptedRunnableExecution.run(InterceptedExecution.java:118) at org.apache.cassandra.simulator.systems.InterceptingExecutor$InterceptingPooledExecutor$WaitingThread.lambda$new$1(InterceptingExecutor.java:318) at org.apache.cassandra.simulator.systems.InterceptingExecutor$InterceptingPooledExecutor$WaitingThread$$Lambda$5312/0x0000000841aa2c40.run(Unknown Source) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) WARN [isolatedExecutor:2] node1 2024-05-29 01:39:59,513 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:1] node2 2024-05-29 01:39:59,513 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:1] node3 2024-05-29 01:39:59,513 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:1] node4 2024-05-29 01:39:59,515 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:1] node5 2024-05-29 01:39:59,516 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:2] node6 2024-05-29 01:39:59,516 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:2] node7 2024-05-29 01:39:59,516 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:1] node8 2024-05-29 01:39:59,517 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:1] node9 2024-05-29 01:39:59,517 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:2] node10 2024-05-29 01:39:59,517 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:1] node11 2024-05-29 01:39:59,517 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:1] node12 2024-05-29 01:39:59,518 HarrySimulatorTest.java:337 - Epoch{epoch=51} Marking 4 as decommissioned ERROR [isolatedExecutor:2] node4 2024-05-29 01:39:59,518 HarrySimulatorTest.java:689 - Node 4 state (LEAVING) is not as expected LEFT ERROR [isolatedExecutor:2] <main> 2024-05-29 01:39:59,518 HarrySimulatorTest.java:980 - Caught an exception, going to halt java.lang.OutOfMemoryError: Requested array size exceeds VM limit WARN [main] <main> 2024-05-29 01:39:59,519 HarrySimulatorTest.java:781 - Starting validation of 4 written partitions. Highest LTS is 4. Ring view: SimulatedNodeState{ring=[dc0-/127.0.0.1:7012@-7686143364045646507, dc0-/127.0.0.2:7012@-6148914691236517207, dc0-/127.0.0.3:7012@-4611686018427387907, dc1-/127.0.0.5:7012@-1537228672809129307, dc1-/127.0.0.6:7012@-7, dc1-/127.0.0.7:7012@1537228672809129293, dc1-/127.0.0.8:7012@3507966432223637333, dc2-/127.0.0.12:7012@3890885889187058965, dc2-/127.0.0.9:7012@4611686018427387893, dc2-/127.0.0.10:7012@6148914691236517193, dc2-/127.0.0.11:7012@7686143364045646493]} WARN [main] <main> 2024-05-29 01:39:59,574 HarrySimulatorTest.java:347 - Finished decommission of 4 and data validation! ERROR [isolatedExecutor:2] node8 2024-05-29 01:39:59,675 HarrySimulatorTest.java:649 - Could not decommission. Interrupting simulation. java.lang.IllegalStateException: Can not commit transformation: "INVALID"(Rejecting this plan as it interacts with a range locked by Key{key=Epoch{epoch=50}} (locked: LockedRanges{lastModified=Epoch{epoch=50}, locked={Key{key=Epoch{epoch=50}}=AffectedRangesImpl{map={ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=2}=[(6148914691236517193,6457576751661366678], (4611686018427387893,6148914691236517193], (6148914691236517193,7686143364045646493]], ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=3}=[(6148914691236517193,6457576751661366678], (4611686018427387893,6148914691236517193], (3890885889187058965,4611686018427387893], (6148914691236517193,7686143364045646493]], ReplicationParams{class=org.apache.cassandra.locator.NetworkTopologyStrategy, dc0=3, dc2=3, dc1=3}=[(6148914691236517193,6457576751661366678], (-7,1537228672809129293], (4611686018427387893,6148914691236517193], (-7686143364045646507,-6148914691236517207], (3507966432223637333,3890885889187058965], (3890885889187058965,4611686018427387893], (-1537228672809129307,-7], (6148914691236517193,7686143364045646493], (-4611686018427387907,-1537228672809129307], (1537228672809129293,3507966432223637333], (-6148914691236517207,-4611686018427387907]]}}}}, new: AffectedRangesImpl{map={ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=2}=[(-7,1537228672809129293], (1537228672809129293,3890885889187058965], (1537228672809129293,3507966432223637333]], ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=3}=[(-7,1537228672809129293], (-1537228672809129307,-7], (1537228672809129293,3890885889187058965], (1537228672809129293,3507966432223637333]], ReplicationParams{class=org.apache.cassandra.locator.NetworkTopologyStrategy, dc0=3, dc2=3, dc1=3}=[(-7,1537228672809129293], (-1537228672809129307,-7], (1537228672809129293,3890885889187058965], (1537228672809129293,3507966432223637333]]}})). at org.apache.cassandra.tcm.ClusterMetadataService.lambda$commit$6(ClusterMetadataService.java:479) at org.apache.cassandra.tcm.ClusterMetadataService.commit(ClusterMetadataService.java:523) at org.apache.cassandra.tcm.ClusterMetadataService.commit(ClusterMetadataService.java:476) at org.apache.cassandra.tcm.sequences.SingleNodeSequences.decommission(SingleNodeSequences.java:79) at org.apache.cassandra.simulator.test.HarrySimulatorTest.lambda$decommission$6ea5f628$1(HarrySimulatorTest.java:645) at org.apache.cassandra.simulator.systems.InterceptedExecution$InterceptedRunnableExecution.run(InterceptedExecution.java:118) at org.apache.cassandra.simulator.systems.InterceptingExecutor$InterceptingPooledExecutor$WaitingThread.lambda$new$1(InterceptingExecutor.java:318) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) ERROR [isolatedExecutor:2] <main> 2024-05-29 01:39:59,677 HarrySimulatorTest.java:980 - Caught an exception, going to halt java.lang.OutOfMemoryError: Requested array size exceeds VM limit WARN [isolatedExecutor:1] node1 2024-05-29 01:39:59,677 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:2] node2 2024-05-29 01:39:59,677 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:2] node3 2024-05-29 01:39:59,677 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:1] node4 2024-05-29 01:39:59,678 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:1] node5 2024-05-29 01:39:59,678 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:1] node6 2024-05-29 01:39:59,678 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:1] node7 2024-05-29 01:39:59,678 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:1] node8 2024-05-29 01:39:59,679 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:2] node9 2024-05-29 01:39:59,679 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:1] node10 2024-05-29 01:39:59,679 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:1] node11 2024-05-29 01:39:59,679 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:1] node12 2024-05-29 01:39:59,680 HarrySimulatorTest.java:337 - Epoch{epoch=51} Marking 8 as decommissioned ERROR [isolatedExecutor:2] node8 2024-05-29 01:39:59,680 HarrySimulatorTest.java:689 - Node 8 state (JOINED) is not as expected LEFT ERROR [isolatedExecutor:2] <main> 2024-05-29 01:39:59,680 HarrySimulatorTest.java:980 - Caught an exception, going to halt java.lang.OutOfMemoryError: Requested array size exceeds VM limit WARN [main] <main> 2024-05-29 01:39:59,680 HarrySimulatorTest.java:781 - Starting validation of 5 written partitions. Highest LTS is 5. Ring view: SimulatedNodeState{ring=[dc0-/127.0.0.1:7012@-7686143364045646507, dc0-/127.0.0.2:7012@-6148914691236517207, dc0-/127.0.0.3:7012@-4611686018427387907, dc1-/127.0.0.5:7012@-1537228672809129307, dc1-/127.0.0.6:7012@-7, dc1-/127.0.0.7:7012@1537228672809129293, dc2-/127.0.0.12:7012@3890885889187058965, dc2-/127.0.0.9:7012@4611686018427387893, dc2-/127.0.0.10:7012@6148914691236517193, dc2-/127.0.0.11:7012@7686143364045646493]} WARN [main] <main> 2024-05-29 01:39:59,704 HarrySimulatorTest.java:347 - Finished decommission of 8 and data validation! WARN [InternalMetadataStage:1] node12 2024-05-29 01:39:59,806 RemoteProcessor.java:238 - Got error from /127.0.0.10:7012: TIMEOUT when sending TCM_COMMIT_REQ, retrying on CandidateIterator{candidates=[/127.0.0.2:7012, /127.0.0.1:7012, /127.0.0.5:7012, /127.0.0.9:7012, /127.0.0.6:7012, /127.0.0.10:7012], checkLive=true} ERROR [isolatedExecutor:2] node12 2024-05-29 01:39:59,834 HarrySimulatorTest.java:649 - Could not decommission. Interrupting simulation. java.lang.IllegalStateException: Can not commit transformation: "INVALID"(Rejecting this plan as it interacts with a range locked by Key{key=Epoch{epoch=50}} (locked: LockedRanges{lastModified=Epoch{epoch=50}, locked={Key{key=Epoch{epoch=50}}=AffectedRangesImpl{map={ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=2}=[(6148914691236517193,6457576751661366678], (4611686018427387893,6148914691236517193], (6148914691236517193,7686143364045646493]], ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=3}=[(6148914691236517193,6457576751661366678], (4611686018427387893,6148914691236517193], (3890885889187058965,4611686018427387893], (6148914691236517193,7686143364045646493]], ReplicationParams{class=org.apache.cassandra.locator.NetworkTopologyStrategy, dc0=3, dc2=3, dc1=3}=[(6148914691236517193,6457576751661366678], (-7,1537228672809129293], (4611686018427387893,6148914691236517193], (-7686143364045646507,-6148914691236517207], (3507966432223637333,3890885889187058965], (3890885889187058965,4611686018427387893], (-1537228672809129307,-7], (6148914691236517193,7686143364045646493], (-4611686018427387907,-1537228672809129307], (1537228672809129293,3507966432223637333], (-6148914691236517207,-4611686018427387907]]}}}}, new: AffectedRangesImpl{map={ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=2}=[(3507966432223637333,3890885889187058965], (3507966432223637333,4611686018427387893], (1537228672809129293,3507966432223637333]], ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=3}=[(-7,1537228672809129293], (3507966432223637333,3890885889187058965], (3507966432223637333,4611686018427387893], (1537228672809129293,3507966432223637333]], ReplicationParams{class=org.apache.cassandra.locator.NetworkTopologyStrategy, dc0=3, dc2=3, dc1=3}=[(6148914691236517193,6457576751661366678], (-9223372036854775808,-7686143364045646507], (-7686143364045646507,-6148914691236517207], (7686143364045646493,-9223372036854775808], (3507966432223637333,4611686018427387893], (-4611686018427387907,-1537228672809129307], (4611686018427387893,6148914691236517193], (-7,1537228672809129293], (6457576751661366678,7686143364045646493], (3507966432223637333,3890885889187058965], (-1537228672809129307,-7], (1537228672809129293,3507966432223637333], (-6148914691236517207,-4611686018427387907]]}})). at org.apache.cassandra.tcm.ClusterMetadataService.lambda$commit$6(ClusterMetadataService.java:479) at org.apache.cassandra.tcm.ClusterMetadataService.commit(ClusterMetadataService.java:523) at org.apache.cassandra.tcm.ClusterMetadataService.commit(ClusterMetadataService.java:476) at org.apache.cassandra.tcm.sequences.SingleNodeSequences.decommission(SingleNodeSequences.java:79) at org.apache.cassandra.simulator.test.HarrySimulatorTest.lambda$decommission$6ea5f628$1(HarrySimulatorTest.java:645) at org.apache.cassandra.simulator.systems.InterceptedExecution$InterceptedRunnableExecution.run(InterceptedExecution.java:118) at org.apache.cassandra.simulator.systems.InterceptingExecutor$InterceptingPooledExecutor$WaitingThread.lambda$new$1(InterceptingExecutor.java:318) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) ERROR [isolatedExecutor:2] <main> 2024-05-29 01:39:59,835 HarrySimulatorTest.java:980 - Caught an exception, going to halt java.lang.OutOfMemoryError: Requested array size exceeds VM limit WARN [isolatedExecutor:1] node1 2024-05-29 01:39:59,836 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:1] node2 2024-05-29 01:39:59,836 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:1] node3 2024-05-29 01:39:59,836 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:2] node4 2024-05-29 01:39:59,836 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:1] node5 2024-05-29 01:39:59,837 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:2] node6 2024-05-29 01:39:59,837 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:1] node7 2024-05-29 01:39:59,837 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:1] node8 2024-05-29 01:39:59,837 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:1] node9 2024-05-29 01:39:59,838 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:2] node10 2024-05-29 01:39:59,838 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:1] node11 2024-05-29 01:39:59,838 HarrySimulatorTest.java:337 - Epoch{epoch=51} WARN [isolatedExecutor:1] node12 2024-05-29 01:39:59,838 HarrySimulatorTest.java:337 - Epoch{epoch=51} Marking 12 as decommissioned ERROR [isolatedExecutor:2] node12 2024-05-29 01:39:59,839 HarrySimulatorTest.java:689 - Node 12 state (JOINED) is not as expected LEFT ERROR [isolatedExecutor:2] <main> 2024-05-29 01:39:59,842 HarrySimulatorTest.java:980 - Caught an exception, going to halt java.lang.OutOfMemoryError: Requested array size exceeds VM limit WARN [main] <main> 2024-05-29 01:39:59,842 HarrySimulatorTest.java:781 - Starting validation of 6 written partitions. Highest LTS is 6. Ring view: SimulatedNodeState{ring=[dc0-/127.0.0.1:7012@-7686143364045646507, dc0-/127.0.0.2:7012@-6148914691236517207, dc0-/127.0.0.3:7012@-4611686018427387907, dc1-/127.0.0.5:7012@-1537228672809129307, dc1-/127.0.0.6:7012@-7, dc1-/127.0.0.7:7012@1537228672809129293, dc2-/127.0.0.9:7012@4611686018427387893, dc2-/127.0.0.10:7012@6148914691236517193, dc2-/127.0.0.11:7012@7686143364045646493]} WARN [main] <main> 2024-05-29 01:39:59,876 HarrySimulatorTest.java:347 - Finished decommission of 12 and data validation! WARN [main] <main> 2024-05-29 01:39:59,876 HarrySimulatorTest.java:350 - Finished! ERROR [HintsWriteExecutor:1] <main> 2024-05-29 01:40:05,838 HarrySimulatorTest.java:980 - Caught an exception, going to halt java.lang.OutOfMemoryError: Direct buffer memory at java.base/java.nio.Bits.reserveMemory(Bits.java:175) at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) at org.apache.cassandra.hints.HintsBuffer.create(HintsBuffer.java:77) at org.apache.cassandra.hints.HintsBufferPool.createBuffer(HintsBufferPool.java:133) at org.apache.cassandra.hints.HintsBufferPool.initializeCurrentBuffer(HintsBufferPool.java:104) at org.apache.cassandra.hints.HintsBufferPool.currentBuffer(HintsBufferPool.java:96) at org.apache.cassandra.hints.HintsWriteExecutor$FlushBufferPoolTask.run(HintsWriteExecutor.java:168) at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96) at org.apache.cassandra.concurrent.SyncFutureTask.run(SyncFutureTask.java:68) at org.apache.cassandra.simulator.systems.SimulatedExecution$NoIntercept$1Run.run(SimulatedExecution.java:82) at org.apache.cassandra.simulator.systems.InterceptingExecutor$AbstractSingleThreadedExecutorPlus.lambda$new$0(InterceptingExecutor.java:585) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) ERROR [isolatedExecutor:1] <main> 2024-05-29 01:40:06,146 HarrySimulatorTest.java:980 - Caught an exception, going to halt java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Direct buffer memory at org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:79) at org.apache.cassandra.distributed.impl.Instance.lambda$shutdown$47(Instance.java:985) at org.apache.cassandra.distributed.impl.IsolatedExecutor.lambda$async$10(IsolatedExecutor.java:156) at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96) at org.apache.cassandra.concurrent.SyncFutureTask.run(SyncFutureTask.java:68) at org.apache.cassandra.simulator.systems.SimulatedExecution$NoIntercept$1Run.run(SimulatedExecution.java:82) at org.apache.cassandra.simulator.systems.InterceptingExecutor$InterceptingPooledExecutor$WaitingThread.lambda$new$1(InterceptingExecutor.java:318) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Direct buffer memory at org.apache.cassandra.utils.concurrent.AbstractFuture.getWhenDone(AbstractFuture.java:239) at org.apache.cassandra.utils.concurrent.AbstractFuture.get(AbstractFuture.java:246) at org.apache.cassandra.hints.HintsService.shutdownBlocking(HintsService.java:281) at org.apache.cassandra.distributed.impl.Instance.lambda$shutdown$17(Instance.java:913) at org.apache.cassandra.distributed.impl.Instance.lambda$parallelRun$53(Instance.java:1209) ... 5 common frames omitted Caused by: java.lang.OutOfMemoryError: Direct buffer memory at java.base/java.nio.Bits.reserveMemory(Bits.java:175) at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) at org.apache.cassandra.hints.HintsBuffer.create(HintsBuffer.java:77) at org.apache.cassandra.hints.HintsBufferPool.createBuffer(HintsBufferPool.java:133) at org.apache.cassandra.hints.HintsBufferPool.initializeCurrentBuffer(HintsBufferPool.java:104) at org.apache.cassandra.hints.HintsBufferPool.currentBuffer(HintsBufferPool.java:96) at org.apache.cassandra.hints.HintsWriteExecutor$FlushBufferPoolTask.run(HintsWriteExecutor.java:168) at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96) at org.apache.cassandra.concurrent.SyncFutureTask.run(SyncFutureTask.java:68) at org.apache.cassandra.simulator.systems.SimulatedExecution$NoIntercept$1Run.run(SimulatedExecution.java:82) at org.apache.cassandra.simulator.systems.InterceptingExecutor$AbstractSingleThreadedExecutorPlus.lambda$new$0(InterceptingExecutor.java:585) ... 2 common frames omitted ERROR [isolatedExecutor:2] <main> 2024-05-29 01:40:06,148 HarrySimulatorTest.java:980 - Caught an exception, going to halt java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Direct buffer memory at org.apache.cassandra.utils.concurrent.AbstractFuture.getWhenDone(AbstractFuture.java:239) at org.apache.cassandra.utils.concurrent.AbstractFuture.get(AbstractFuture.java:246) at org.apache.cassandra.distributed.impl.Instance.lambda$shutdown$48(Instance.java:991) at org.apache.cassandra.concurrent.SyncFutureTask.run(SyncFutureTask.java:68) at org.apache.cassandra.simulator.systems.SimulatedExecution$NoIntercept$1Run.run(SimulatedExecution.java:82) at org.apache.cassandra.simulator.systems.InterceptingExecutor$InterceptingPooledExecutor$WaitingThread.lambda$new$1(InterceptingExecutor.java:318) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Direct buffer memory at org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:79) at org.apache.cassandra.distributed.impl.Instance.lambda$shutdown$47(Instance.java:985) at org.apache.cassandra.distributed.impl.IsolatedExecutor.lambda$async$10(IsolatedExecutor.java:156) at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96) ... 5 common frames omitted Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Direct buffer memory at org.apache.cassandra.utils.concurrent.AbstractFuture.getWhenDone(AbstractFuture.java:239) at org.apache.cassandra.utils.concurrent.AbstractFuture.get(AbstractFuture.java:246) at org.apache.cassandra.hints.HintsService.shutdownBlocking(HintsService.java:281) at org.apache.cassandra.distributed.impl.Instance.lambda$shutdown$17(Instance.java:913) at org.apache.cassandra.distributed.impl.Instance.lambda$parallelRun$53(Instance.java:1209) ... 5 common frames omitted Caused by: java.lang.OutOfMemoryError: Direct buffer memory at java.base/java.nio.Bits.reserveMemory(Bits.java:175) at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) at org.apache.cassandra.hints.HintsBuffer.create(HintsBuffer.java:77) at org.apache.cassandra.hints.HintsBufferPool.createBuffer(HintsBufferPool.java:133) at org.apache.cassandra.hints.HintsBufferPool.initializeCurrentBuffer(HintsBufferPool.java:104) at org.apache.cassandra.hints.HintsBufferPool.currentBuffer(HintsBufferPool.java:96) at org.apache.cassandra.hints.HintsWriteExecutor$FlushBufferPoolTask.run(HintsWriteExecutor.java:168) at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96) at org.apache.cassandra.concurrent.SyncFutureTask.run(SyncFutureTask.java:68) at org.apache.cassandra.simulator.systems.SimulatedExecution$NoIntercept$1Run.run(SimulatedExecution.java:82) at org.apache.cassandra.simulator.systems.InterceptingExecutor$AbstractSingleThreadedExecutorPlus.lambda$new$0(InterceptingExecutor.java:585) ... 2 common frames omitted