org.apache.flink.table.runtime.operators.sort

Class BinaryExternalSorter

java.lang.Object

org.apache.flink.table.runtime.operators.sort.BinaryExternalSorter

All Implemented Interfaces:

Closeable, AutoCloseable, Sorter<BinaryRowData>, CloseableInputProvider<BinaryRowData>

public class BinaryExternalSorter
extends Object
implements Sorter<BinaryRowData>

The BinaryExternalSorter is a full fledged sorter for binary format. It implements a multi-way merge sort. Internally, it has three asynchronous threads (sort, spill, merger) which communicate through a set of blocking circularQueues, forming a closed loop. Memory is allocated using the MemoryManager interface. Thus the component will not exceed the provided memory limits.

Constructor Summary

Constructors

Constructor and Description
BinaryExternalSorter(Object owner, MemoryManager memoryManager, long reservedMemorySize, IOManager ioManager, AbstractRowDataSerializer<RowData> inputSerializer, BinaryRowDataSerializer serializer, NormalizedKeyComputer normalizedKeyComputer, RecordComparator comparator, int maxNumFileHandles, boolean compressionEnabled, int compressionBlockSize, boolean asyncMergeEnabled)
BinaryExternalSorter(Object owner, MemoryManager memoryManager, long reservedMemorySize, IOManager ioManager, AbstractRowDataSerializer<RowData> inputSerializer, BinaryRowDataSerializer serializer, NormalizedKeyComputer normalizedKeyComputer, RecordComparator comparator, int maxNumFileHandles, boolean compressionEnabled, int compressionBlockSize, boolean asyncMergeEnabled, float startSpillingFraction)

Method Summary

Modifier and Type	Method and Description
void	close() Shuts down all the threads initiated by this sorter.
MutableObjectIterator<BinaryRowData>	getIterator() Gets the iterator over this input.
long	getNumSpillFiles()
long	getSpillInBytes()
long	getUsedMemoryInBytes()
void	startThreads() Starts all the threads that are used by this sorter.
void	write(MutableObjectIterator<BinaryRowData> iterator)
void	write(RowData current)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BinaryExternalSorter

public BinaryExternalSorter(Object owner,
                            MemoryManager memoryManager,
                            long reservedMemorySize,
                            IOManager ioManager,
                            AbstractRowDataSerializer<RowData> inputSerializer,
                            BinaryRowDataSerializer serializer,
                            NormalizedKeyComputer normalizedKeyComputer,
                            RecordComparator comparator,
                            int maxNumFileHandles,
                            boolean compressionEnabled,
                            int compressionBlockSize,
                            boolean asyncMergeEnabled)

BinaryExternalSorter

public BinaryExternalSorter(Object owner,
                            MemoryManager memoryManager,
                            long reservedMemorySize,
                            IOManager ioManager,
                            AbstractRowDataSerializer<RowData> inputSerializer,
                            BinaryRowDataSerializer serializer,
                            NormalizedKeyComputer normalizedKeyComputer,
                            RecordComparator comparator,
                            int maxNumFileHandles,
                            boolean compressionEnabled,
                            int compressionBlockSize,
                            boolean asyncMergeEnabled,
                            float startSpillingFraction)

Method Detail

startThreads

public void startThreads()

Starts all the threads that are used by this sorter.

close

public void close()

Shuts down all the threads initiated by this sorter. Also releases all previously allocated memory, if it has not yet been released by the threads, and closes and deletes all channels (removing the temporary files).

The threads are set to exit directly, but depending on their operation, it may take a while to actually happen. The sorting thread will for example not finish before the current batch is sorted. This method attempts to wait for the working thread to exit. If it is however interrupted, the method exits immediately and is not guaranteed how long the threads continue to exist and occupy resources afterwards.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

write

public void write(RowData current)
           throws IOException

Throws:

IOException

write

@VisibleForTesting
public void write(MutableObjectIterator<BinaryRowData> iterator)
                              throws IOException

Throws:

IOException

getIterator

public MutableObjectIterator<BinaryRowData> getIterator()
                                                 throws InterruptedException

Description copied from interface: CloseableInputProvider

Gets the iterator over this input.

Specified by:

getIterator in interface CloseableInputProvider<BinaryRowData>

Returns:

The iterator provided by this iterator provider.

Throws:

InterruptedException

getUsedMemoryInBytes

public long getUsedMemoryInBytes()

getNumSpillFiles

public long getNumSpillFiles()

getSpillInBytes

public long getSpillInBytes()