public class HashBufferAccumulator extends Object implements BufferAccumulator, HashSubpartitionBufferAccumulatorContext
BufferAccumulator
. The BufferAccumulator
receives
the records from TieredStorageProducerClient
and the records will accumulate and
transform to finished buffers. The accumulated buffers will be transferred to the corresponding
tier dynamically.
To avoid the buffer waiting deadlock between the subpartitions, the HashBufferAccumulator
requires at least n buffers (n is the number of subpartitions) to make
sure that each subpartition has at least one buffer to accumulate the receiving data. Once an
accumulated buffer is finished, the buffer will be flushed immediately.
Note that this class need not be thread-safe, because it should only be accessed from the main thread.
Constructor and Description |
---|
HashBufferAccumulator(int numSubpartitions,
int bufferSize,
TieredStorageMemoryManager memoryManager,
boolean isPartialRecordAllowed) |
Modifier and Type | Method and Description |
---|---|
void |
close()
Close the accumulator.
|
void |
flushAccumulatedBuffers(TieredStorageSubpartitionId subpartitionId,
Buffer accumulatedBuffer,
int numRemainingConsecutiveBuffers)
Flush the accumulated
Buffer s of the subpartition. |
void |
receive(ByteBuffer record,
TieredStorageSubpartitionId subpartitionId,
Buffer.DataType dataType,
boolean isBroadcast)
Receives the records from tiered store producer, these records will be accumulated and
transformed into finished buffers.
|
BufferBuilder |
requestBufferBlocking()
Request
BufferBuilder from the BufferPool . |
void |
setup(TriConsumer<TieredStorageSubpartitionId,Buffer,Integer> accumulatedBufferFlusher)
Setup the accumulator.
|
public HashBufferAccumulator(int numSubpartitions, int bufferSize, TieredStorageMemoryManager memoryManager, boolean isPartialRecordAllowed)
public void setup(TriConsumer<TieredStorageSubpartitionId,Buffer,Integer> accumulatedBufferFlusher)
BufferAccumulator
setup
in interface BufferAccumulator
accumulatedBufferFlusher
- accepts the accumulated buffers. The first field is the subpartition id,
the second is the accumulated buffer to flush, and the third is the number of remaining
buffers to be written consecutively to the same segment.public void receive(ByteBuffer record, TieredStorageSubpartitionId subpartitionId, Buffer.DataType dataType, boolean isBroadcast) throws IOException
BufferAccumulator
Note that when isBroadcast is true, for a broadcast-only partition, the subpartitionId value will always be 0. Conversely, for a non-broadcast-only partition, the subpartitionId value will range from 0 to the number of subpartitions.
receive
in interface BufferAccumulator
record
- the received recordsubpartitionId
- the subpartition id of the recorddataType
- the data type of the recordisBroadcast
- whether the record is a broadcast recordIOException
public void close()
BufferAccumulator
close
in interface AutoCloseable
close
in interface BufferAccumulator
public BufferBuilder requestBufferBlocking()
HashSubpartitionBufferAccumulatorContext
BufferBuilder
from the BufferPool
.requestBufferBlocking
in interface HashSubpartitionBufferAccumulatorContext
public void flushAccumulatedBuffers(TieredStorageSubpartitionId subpartitionId, Buffer accumulatedBuffer, int numRemainingConsecutiveBuffers)
HashSubpartitionBufferAccumulatorContext
Buffer
s of the subpartition.flushAccumulatedBuffers
in interface HashSubpartitionBufferAccumulatorContext
subpartitionId
- the subpartition idaccumulatedBuffer
- the accumulated buffernumRemainingConsecutiveBuffers
- number of buffers that would be passed in the following
invocations and should be written to the same segment as this oneCopyright © 2014–2024 The Apache Software Foundation. All rights reserved.