public class HsMemoryDataManager extends Object implements HsSpillingInfoProvider, HsMemoryDataManagerOperation
HsSpillingInfoProvider.ConsumeStatus, HsSpillingInfoProvider.SpillStatus
Constructor and Description |
---|
HsMemoryDataManager(int numSubpartitions,
int bufferSize,
BufferPool bufferPool,
HsSpillingStrategy spillStrategy,
HsFileDataIndex fileDataIndex,
Path dataFilePath,
BufferCompressor bufferCompressor,
long poolSizeCheckInterval) |
Modifier and Type | Method and Description |
---|---|
void |
append(ByteBuffer record,
int targetChannel,
Buffer.DataType dataType)
Append record to
HsMemoryDataManager , It will be managed by HsSubpartitionMemoryDataManager witch it belongs to. |
void |
close()
Close this
HsMemoryDataManager , it means no data can append to memory. |
Deque<BufferIndexAndChannel> |
getBuffersInOrder(int subpartitionId,
HsSpillingInfoProvider.SpillStatus spillStatus,
HsSpillingInfoProvider.ConsumeStatus consumeStatus)
Get all buffers with the expected status from the subpartition.
|
List<Integer> |
getNextBufferIndexToConsume()
Get all downstream next buffer index to consume.
|
int |
getNumSubpartitions()
Get the number of downstream consumers.
|
int |
getNumTotalRequestedBuffers()
Get total number of buffers requested from buffer pool.
|
int |
getNumTotalUnSpillBuffers()
Get total number of not decided to spill buffers.
|
int |
getPoolSize()
Get the current size of buffer pool.
|
void |
markBufferReleasedFromFile(int subpartitionId,
int bufferIndex)
This method is called when buffer should mark as released in
HsFileDataIndex . |
void |
onBufferConsumed(BufferIndexAndChannel consumedBuffer)
This method is called when buffer is consumed.
|
void |
onBufferFinished()
This method is called when buffer is finished.
|
void |
onDataAvailable(int subpartitionId)
This method is called when subpartition data become available.
|
HsDataView |
registerSubpartitionView(int subpartitionId,
HsSubpartitionViewInternalOperations viewOperations)
|
void |
release()
Release this
HsMemoryDataManager , it means all memory taken by this class will
recycle. |
BufferBuilder |
requestBufferFromPool()
Request buffer from buffer pool.
|
void |
setOutputMetrics(HsOutputMetrics metrics) |
public HsMemoryDataManager(int numSubpartitions, int bufferSize, BufferPool bufferPool, HsSpillingStrategy spillStrategy, HsFileDataIndex fileDataIndex, Path dataFilePath, BufferCompressor bufferCompressor, long poolSizeCheckInterval) throws IOException
IOException
public void append(ByteBuffer record, int targetChannel, Buffer.DataType dataType) throws IOException
HsMemoryDataManager
, It will be managed by HsSubpartitionMemoryDataManager
witch it belongs to.record
- to be managed by this class.targetChannel
- target subpartition of this record.dataType
- the type of this record. In other words, is it data or event.IOException
public HsDataView registerSubpartitionView(int subpartitionId, HsSubpartitionViewInternalOperations viewOperations)
HsSubpartitionViewInternalOperations
to subpartitionViewOperationsMap
. It is used to obtain the consumption progress of the
subpartition.public void close()
HsMemoryDataManager
, it means no data can append to memory.public void release()
HsMemoryDataManager
, it means all memory taken by this class will
recycle.public void setOutputMetrics(HsOutputMetrics metrics)
public int getPoolSize()
HsSpillingInfoProvider
getPoolSize
in interface HsSpillingInfoProvider
public int getNumSubpartitions()
HsSpillingInfoProvider
getNumSubpartitions
in interface HsSpillingInfoProvider
public int getNumTotalRequestedBuffers()
HsSpillingInfoProvider
getNumTotalRequestedBuffers
in interface HsSpillingInfoProvider
public int getNumTotalUnSpillBuffers()
HsSpillingInfoProvider
getNumTotalUnSpillBuffers
in interface HsSpillingInfoProvider
public Deque<BufferIndexAndChannel> getBuffersInOrder(int subpartitionId, HsSpillingInfoProvider.SpillStatus spillStatus, HsSpillingInfoProvider.ConsumeStatus consumeStatus)
HsSpillingInfoProvider
getBuffersInOrder
in interface HsSpillingInfoProvider
subpartitionId
- target buffers belong to.spillStatus
- expected buffer spill status.consumeStatus
- expected buffer consume status.public List<Integer> getNextBufferIndexToConsume()
HsSpillingInfoProvider
getNextBufferIndexToConsume
in interface HsSpillingInfoProvider
public void markBufferReleasedFromFile(int subpartitionId, int bufferIndex)
HsMemoryDataManagerOperation
HsFileDataIndex
.markBufferReleasedFromFile
in interface HsMemoryDataManagerOperation
subpartitionId
- the subpartition that target buffer belong to.bufferIndex
- index of buffer to mark as released.public BufferBuilder requestBufferFromPool() throws InterruptedException
HsMemoryDataManagerOperation
requestBufferFromPool
in interface HsMemoryDataManagerOperation
InterruptedException
public void onBufferConsumed(BufferIndexAndChannel consumedBuffer)
HsMemoryDataManagerOperation
onBufferConsumed
in interface HsMemoryDataManagerOperation
consumedBuffer
- target buffer to mark as consumed.public void onBufferFinished()
HsMemoryDataManagerOperation
onBufferFinished
in interface HsMemoryDataManagerOperation
public void onDataAvailable(int subpartitionId)
HsMemoryDataManagerOperation
onDataAvailable
in interface HsMemoryDataManagerOperation
subpartitionId
- the subpartition need notify data available.Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.