public class SortBufferAccumulator extends Object implements BufferAccumulator
BufferAccumulator
. The BufferAccumulator
receives the records from TieredStorageProducerClient
and the records will accumulate and
transform to finished buffers. The accumulated buffers will be transferred to the corresponding
tier dynamically.
The SortBufferAccumulator
can help use less buffers to accumulate data, which
decouples the buffer usage with the number of parallelism. The number of buffers used by the
SortBufferAccumulator
will be numExpectedBuffers at most. Once the DataBuffer
is
full, or switching from broadcast to non-broadcast(or vice versa), the buffer in the sort buffer
will be flushed to the tiers.
Note that this class need not be thread-safe, because it should only be accessed from the main thread.
Constructor and Description |
---|
SortBufferAccumulator(int numSubpartitions,
int numExpectedBuffers,
int bufferSizeBytes,
long poolSizeCheckInterval,
TieredStorageMemoryManager memoryManager,
boolean isPartialRecordAllowed) |
Modifier and Type | Method and Description |
---|---|
void |
close()
Close the accumulator.
|
void |
receive(ByteBuffer record,
TieredStorageSubpartitionId subpartitionId,
Buffer.DataType dataType,
boolean isBroadcast)
Receives the records from tiered store producer, these records will be accumulated and
transformed into finished buffers.
|
void |
setup(TriConsumer<TieredStorageSubpartitionId,Buffer,Integer> bufferFlusher)
Setup the accumulator.
|
public SortBufferAccumulator(int numSubpartitions, int numExpectedBuffers, int bufferSizeBytes, long poolSizeCheckInterval, TieredStorageMemoryManager memoryManager, boolean isPartialRecordAllowed)
public void setup(TriConsumer<TieredStorageSubpartitionId,Buffer,Integer> bufferFlusher)
BufferAccumulator
setup
in interface BufferAccumulator
bufferFlusher
- accepts the accumulated buffers. The first field is the subpartition id,
the second is the accumulated buffer to flush, and the third is the number of remaining
buffers to be written consecutively to the same segment.public void receive(ByteBuffer record, TieredStorageSubpartitionId subpartitionId, Buffer.DataType dataType, boolean isBroadcast) throws IOException
BufferAccumulator
Note that when isBroadcast is true, for a broadcast-only partition, the subpartitionId value will always be 0. Conversely, for a non-broadcast-only partition, the subpartitionId value will range from 0 to the number of subpartitions.
receive
in interface BufferAccumulator
record
- the received recordsubpartitionId
- the subpartition id of the recorddataType
- the data type of the recordisBroadcast
- whether the record is a broadcast recordIOException
public void close()
BufferAccumulator
close
in interface AutoCloseable
close
in interface BufferAccumulator
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.