public class CombiningUnilateralSortMerger<E> extends UnilateralSortMerger<E>
CombiningUnilateralSortMerger
logic is factored into four threads (read, sort, combine,
spill) which communicate through a set of blocking queues (forming a closed loop).
Memory is allocated using the MemoryManager
interface. Thus the component will most likely not exceed the
user-provided memory limits.Modifier and Type | Class and Description |
---|---|
protected class |
CombiningUnilateralSortMerger.CombiningSpillingThread |
UnilateralSortMerger.ChannelWithBlockCount, UnilateralSortMerger.CircularElement<E>, UnilateralSortMerger.CircularQueues<E>, UnilateralSortMerger.ReadingThread<E>, UnilateralSortMerger.SortingThread<E>, UnilateralSortMerger.SpillingThread, UnilateralSortMerger.ThreadBase<E>
closed, iterator, iteratorException, iteratorLock, MAX_NUM_WRITE_BUFFERS, memoryManager, MIN_NUM_SORT_MEM_SEGMENTS, MIN_NUM_WRITE_BUFFERS, objectReuseEnabled, sortReadMemory, writeMemory
Constructor and Description |
---|
CombiningUnilateralSortMerger(GroupCombineFunction<E,E> combineStub,
MemoryManager memoryManager,
IOManager ioManager,
MutableObjectIterator<E> input,
AbstractInvokable parentTask,
TypeSerializerFactory<E> serializerFactory,
TypeComparator<E> comparator,
double memoryFraction,
int maxNumFileHandles,
float startSpillingFraction,
boolean handleLargeRecords,
boolean objectReuseEnabled)
Creates a new sorter that reads the data from a given reader and provides an iterator returning that
data in a sorted manner.
|
CombiningUnilateralSortMerger(GroupCombineFunction<E,E> combineStub,
MemoryManager memoryManager,
IOManager ioManager,
MutableObjectIterator<E> input,
AbstractInvokable parentTask,
TypeSerializerFactory<E> serializerFactory,
TypeComparator<E> comparator,
double memoryFraction,
int numSortBuffers,
int maxNumFileHandles,
float startSpillingFraction,
boolean handleLargeRecords,
boolean objectReuseEnabled)
Creates a new sorter that reads the data from a given reader and provides an iterator returning that
data in a sorted manner.
|
Modifier and Type | Method and Description |
---|---|
protected UnilateralSortMerger.ThreadBase<E> |
getSpillingThread(ExceptionHandler<IOException> exceptionHandler,
UnilateralSortMerger.CircularQueues<E> queues,
AbstractInvokable parentTask,
MemoryManager memoryManager,
IOManager ioManager,
TypeSerializerFactory<E> serializerFactory,
TypeComparator<E> comparator,
List<MemorySegment> sortReadMemory,
List<MemorySegment> writeMemory,
int maxFileHandles) |
void |
setUdfConfiguration(Configuration config) |
close, endMarker, getIterator, getReadingThread, getSortingThread, setResultIterator, setResultIteratorException, spillingMarker, startThreads
public CombiningUnilateralSortMerger(GroupCombineFunction<E,E> combineStub, MemoryManager memoryManager, IOManager ioManager, MutableObjectIterator<E> input, AbstractInvokable parentTask, TypeSerializerFactory<E> serializerFactory, TypeComparator<E> comparator, double memoryFraction, int maxNumFileHandles, float startSpillingFraction, boolean handleLargeRecords, boolean objectReuseEnabled) throws IOException, MemoryAllocationException
combineStub
- The stub used to combine values with the same key.memoryManager
- The memory manager from which to allocate the memory.ioManager
- The I/O manager, which is used to write temporary files to disk.input
- The input that is sorted by this sorter.parentTask
- The parent task, which owns all resources used by this sorter.serializerFactory
- The type serializer.comparator
- The type comparator establishing the order relation.memoryFraction
- The fraction of memory dedicated to sorting, merging and I/O.maxNumFileHandles
- The maximum number of files to be merged at once.startSpillingFraction
- The faction of the buffers that have to be filled before the spilling thread
actually begins spilling data to disk.IOException
- Thrown, if an error occurs initializing the resources for external sorting.MemoryAllocationException
- Thrown, if not enough memory can be obtained from the memory manager to
perform the sort.public CombiningUnilateralSortMerger(GroupCombineFunction<E,E> combineStub, MemoryManager memoryManager, IOManager ioManager, MutableObjectIterator<E> input, AbstractInvokable parentTask, TypeSerializerFactory<E> serializerFactory, TypeComparator<E> comparator, double memoryFraction, int numSortBuffers, int maxNumFileHandles, float startSpillingFraction, boolean handleLargeRecords, boolean objectReuseEnabled) throws IOException, MemoryAllocationException
combineStub
- The stub used to combine values with the same key.memoryManager
- The memory manager from which to allocate the memory.ioManager
- The I/O manager, which is used to write temporary files to disk.input
- The input that is sorted by this sorter.parentTask
- The parent task, which owns all resources used by this sorter.serializerFactory
- The type serializer.comparator
- The type comparator establishing the order relation.memoryFraction
- The fraction of memory dedicated to sorting, merging and I/O.numSortBuffers
- The number of distinct buffers to use creation of the initial runs.maxNumFileHandles
- The maximum number of files to be merged at once.startSpillingFraction
- The faction of the buffers that have to be filled before the spilling thread
actually begins spilling data to disk.IOException
- Thrown, if an error occurs initializing the resources for external sorting.MemoryAllocationException
- Thrown, if not enough memory can be obtained from the memory manager to
perform the sort.public void setUdfConfiguration(Configuration config)
protected UnilateralSortMerger.ThreadBase<E> getSpillingThread(ExceptionHandler<IOException> exceptionHandler, UnilateralSortMerger.CircularQueues<E> queues, AbstractInvokable parentTask, MemoryManager memoryManager, IOManager ioManager, TypeSerializerFactory<E> serializerFactory, TypeComparator<E> comparator, List<MemorySegment> sortReadMemory, List<MemorySegment> writeMemory, int maxFileHandles)
getSpillingThread
in class UnilateralSortMerger<E>
Copyright © 2014–2020 The Apache Software Foundation. All rights reserved.