public class ProducerMergedPartitionFileReader extends Object implements PartitionFileReader
PartitionFileReader
with producer-merge mode. In this mode, the
shuffle data is written in the producer side, the consumer side need to read multiple producers
to get its partition data.
Note that one partition file may contain the data of multiple subpartitions.
Modifier and Type | Class and Description |
---|---|
static class |
ProducerMergedPartitionFileReader.ProducerMergedReadProgress
The implementation of
PartitionFileReader.ReadProgress mainly includes current
reading offset, end of read offset, etc. |
PartitionFileReader.ReadBufferResult, PartitionFileReader.ReadProgress
Modifier and Type | Method and Description |
---|---|
long |
getPriority(TieredStoragePartitionId partitionId,
TieredStorageSubpartitionId subpartitionId,
int segmentId,
int bufferIndex,
PartitionFileReader.ReadProgress readProgress)
Get the priority for reading a particular buffer from the partitioned file.
|
PartitionFileReader.ReadBufferResult |
readBuffer(TieredStoragePartitionId partitionId,
TieredStorageSubpartitionId subpartitionId,
int segmentId,
int bufferIndex,
MemorySegment memorySegment,
BufferRecycler recycler,
PartitionFileReader.ReadProgress readProgress,
CompositeBuffer partialBuffer)
Read a buffer from the partition file.
|
void |
release()
Release the
PartitionFileReader . |
public PartitionFileReader.ReadBufferResult readBuffer(TieredStoragePartitionId partitionId, TieredStorageSubpartitionId subpartitionId, int segmentId, int bufferIndex, MemorySegment memorySegment, BufferRecycler recycler, @Nullable PartitionFileReader.ReadProgress readProgress, @Nullable CompositeBuffer partialBuffer) throws IOException
PartitionFileReader
readBuffer
in interface PartitionFileReader
partitionId
- the partition id of the buffersubpartitionId
- the subpartition id of the buffersegmentId
- the segment id of the bufferbufferIndex
- the index of buffermemorySegment
- the empty buffer to store the read bufferrecycler
- the buffer recyclerreadProgress
- the current read progress. The progress comes from the previous
ReadBufferResult. Note that the read progress should be implemented and provided by
Flink, and it should be directly tied to the file format. The field can be null if the
current file reader has no the read progresspartialBuffer
- the previous partial buffer. The partial buffer is not null only when
the last read has a partial buffer, it will construct a full buffer during the read
process.IOException
- if an error happens.public long getPriority(TieredStoragePartitionId partitionId, TieredStorageSubpartitionId subpartitionId, int segmentId, int bufferIndex, @Nullable PartitionFileReader.ReadProgress readProgress) throws IOException
PartitionFileReader
Depending on the partition file implementation, following the suggestions should typically result in better performance and efficiency. This can be achieved by e.g. choosing preloaded data over others, optimizing the order of disk access to be more sequential, etc.
Note: Priorities are suggestions rather than a requirements. The caller can still read data in whichever order it wants.
getPriority
in interface PartitionFileReader
partitionId
- the partition id of the buffersubpartitionId
- the subpartition id of the buffersegmentId
- the segment id of the bufferbufferIndex
- the index of bufferreadProgress
- the current read progress. The progress comes from the previous
ReadBufferResult. Note that the read progress should be implemented and provided by
Flink, and it should be directly tied to the file format. The field can be null if the
current file reader has no the read progressPartitionFileReader
.IOException
- if an error happens.public void release()
PartitionFileReader
PartitionFileReader
.release
in interface PartitionFileReader
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.