org.apache.flink.runtime.io.disk.iomanager

Class IOManagerAsync

java.lang.Object

org.apache.flink.runtime.io.disk.iomanager.IOManager

org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync

All Implemented Interfaces:

Thread.UncaughtExceptionHandler

public class IOManagerAsync
extends IOManager
implements Thread.UncaughtExceptionHandler

A version of the IOManager that uses asynchronous I/O.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.runtime.io.disk.iomanager.IOManager

IOManager.IOMode

Field Summary

Fields inherited from class org.apache.flink.runtime.io.disk.iomanager.IOManager

LOG

Constructor Summary

Constructors

Constructor and Description
IOManagerAsync() Constructs a new asynchronous I/O manger, writing files to the system 's temp directory.
IOManagerAsync(String tempDir) Constructs a new asynchronous I/O manger, writing file to the given directory.
IOManagerAsync(String[] tempDirs) Constructs a new asynchronous I/O manger, writing file round robin across the given directories.

Method Summary

Modifier and Type	Method and Description
BlockChannelReader<MemorySegment>	createBlockChannelReader(FileIOChannel.ID channelID, LinkedBlockingQueue<MemorySegment> returnQueue) Creates a block channel reader that reads blocks from the given channel.
BlockChannelWriter<MemorySegment>	createBlockChannelWriter(FileIOChannel.ID channelID, LinkedBlockingQueue<MemorySegment> returnQueue) Creates a block channel writer that writes to the given channel.
BlockChannelWriterWithCallback<MemorySegment>	createBlockChannelWriter(FileIOChannel.ID channelID, RequestDoneCallback<MemorySegment> callback) Creates a block channel writer that writes to the given channel.
BufferFileReader	createBufferFileReader(FileIOChannel.ID channelID, RequestDoneCallback<Buffer> callback)
BufferFileSegmentReader	createBufferFileSegmentReader(FileIOChannel.ID channelID, RequestDoneCallback<FileSegment> callback)
BufferFileWriter	createBufferFileWriter(FileIOChannel.ID channelID)
BulkBlockChannelReader	createBulkBlockChannelReader(FileIOChannel.ID channelID, List<MemorySegment> targetSegments, int numBlocks) Creates a block channel reader that reads all blocks from the given channel directly in one bulk.
boolean	isProperlyShutDown() Utility method to check whether the IO manager has been properly shut down.
void	shutdown() Close method.
void	uncaughtException(Thread t, Throwable e)

Methods inherited from class org.apache.flink.runtime.io.disk.iomanager.IOManager

createBlockChannelReader, createBlockChannelWriter, createChannel, createChannelEnumerator, deleteChannel, getNextPathNum, getNumberOfSpillingDirectories, getSpillingDirectories, getSpillingDirectoriesPaths

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

IOManagerAsync

public IOManagerAsync()

Constructs a new asynchronous I/O manger, writing files to the system 's temp directory.

IOManagerAsync

public IOManagerAsync(String tempDir)

Constructs a new asynchronous I/O manger, writing file to the given directory.

Parameters:

tempDir - The directory to write temporary files to.

IOManagerAsync

public IOManagerAsync(String[] tempDirs)

Constructs a new asynchronous I/O manger, writing file round robin across the given directories.

Parameters:

tempDirs - The directories to write temporary files to.

Method Detail

shutdown

public void shutdown()

Close method. Shuts down the reader and writer threads immediately, not waiting for their pending requests to be served. This method waits until the threads have actually ceased their operation.

Overrides:

shutdown in class IOManager

isProperlyShutDown

public boolean isProperlyShutDown()

Utility method to check whether the IO manager has been properly shut down. The IO manager is considered to be properly shut down when it is closed and its threads have ceased operation.

Overrides:

isProperlyShutDown in class IOManager

Returns:

True, if the IO manager has properly shut down, false otherwise.

uncaughtException

public void uncaughtException(Thread t,
                              Throwable e)

Specified by:

uncaughtException in interface Thread.UncaughtExceptionHandler

createBlockChannelWriter

public BlockChannelWriter<MemorySegment> createBlockChannelWriter(FileIOChannel.ID channelID,
                                                                  LinkedBlockingQueue<MemorySegment> returnQueue)
                                                           throws IOException

Description copied from class: IOManager

Creates a block channel writer that writes to the given channel. The writer adds the written segment to the given queue (to allow for asynchronous implementations).

Specified by:

createBlockChannelWriter in class IOManager

Parameters:

channelID - The descriptor for the channel to write to.

returnQueue - The queue to put the written buffers into.

Returns:

A block channel writer that writes to the given channel.

Throws:

IOException - Thrown, if the channel for the writer could not be opened.

createBlockChannelWriter

public BlockChannelWriterWithCallback<MemorySegment> createBlockChannelWriter(FileIOChannel.ID channelID,
                                                                              RequestDoneCallback<MemorySegment> callback)
                                                                       throws IOException

Description copied from class: IOManager

Creates a block channel writer that writes to the given channel. The writer calls the given callback after the I/O operation has been performed (successfully or unsuccessfully), to allow for asynchronous implementations.

Specified by:

createBlockChannelWriter in class IOManager

Parameters:

channelID - The descriptor for the channel to write to.

callback - The callback to be called for

Returns:

A block channel writer that writes to the given channel.

Throws:

IOException - Thrown, if the channel for the writer could not be opened.

createBlockChannelReader

public BlockChannelReader<MemorySegment> createBlockChannelReader(FileIOChannel.ID channelID,
                                                                  LinkedBlockingQueue<MemorySegment> returnQueue)
                                                           throws IOException

Creates a block channel reader that reads blocks from the given channel. The reader reads asynchronously, such that a read request is accepted, carried out at some (close) point in time, and the full segment is pushed to the given queue.

Specified by:

createBlockChannelReader in class IOManager

Parameters:

channelID - The descriptor for the channel to write to.

returnQueue - The queue to put the full buffers into.

Returns:

A block channel reader that reads from the given channel.

Throws:

IOException - Thrown, if the channel for the reader could not be opened.

createBufferFileWriter

public BufferFileWriter createBufferFileWriter(FileIOChannel.ID channelID)
                                        throws IOException

Specified by:

createBufferFileWriter in class IOManager

Throws:

IOException

createBufferFileReader

public BufferFileReader createBufferFileReader(FileIOChannel.ID channelID,
                                               RequestDoneCallback<Buffer> callback)
                                        throws IOException

Specified by:

createBufferFileReader in class IOManager

Throws:

IOException

createBufferFileSegmentReader

public BufferFileSegmentReader createBufferFileSegmentReader(FileIOChannel.ID channelID,
                                                             RequestDoneCallback<FileSegment> callback)
                                                      throws IOException

Specified by:

createBufferFileSegmentReader in class IOManager

Throws:

IOException

createBulkBlockChannelReader

public BulkBlockChannelReader createBulkBlockChannelReader(FileIOChannel.ID channelID,
                                                           List<MemorySegment> targetSegments,
                                                           int numBlocks)
                                                    throws IOException

Creates a block channel reader that reads all blocks from the given channel directly in one bulk. The reader draws segments to read the blocks into from a supplied list, which must contain as many segments as the channel has blocks. After the reader is done, the list with the full segments can be obtained from the reader.

If a channel is not to be read in one bulk, but in multiple smaller batches, a BlockChannelReader should be used.

Specified by:

createBulkBlockChannelReader in class IOManager

Parameters:

channelID - The descriptor for the channel to write to.

targetSegments - The list to take the segments from into which to read the data.

numBlocks - The number of blocks in the channel to read.

Returns:

A block channel reader that reads from the given channel.

Throws:

IOException - Thrown, if the channel for the reader could not be opened.