org.apache.flink.fs.s3.common.writer

Class S3RecoverableFsDataOutputStream

java.lang.Object

java.io.OutputStream

org.apache.flink.core.fs.FSDataOutputStream

org.apache.flink.core.fs.RecoverableFsDataOutputStream

org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream

All Implemented Interfaces:

Closeable, Flushable, AutoCloseable

@PublicEvolving
 @NotThreadSafe
public final class S3RecoverableFsDataOutputStream
extends RecoverableFsDataOutputStream

A RecoverableFsDataOutputStream to S3 that is based on a recoverable multipart upload.

This class is NOT thread-safe. Concurrent writes tho this stream result in corrupt or lost data.

The close() method may be called concurrently when cancelling / shutting down. It will still ensure that local transient resources (like streams and temp files) are cleaned up, but will not touch data previously persisted in S3.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.core.fs.RecoverableFsDataOutputStream

RecoverableFsDataOutputStream.Committer

Method Summary

Modifier and Type	Method and Description
void	close() Closes this stream.
RecoverableFsDataOutputStream.Committer	closeForCommit() Closes the stream, ensuring persistence of all data (similar to FSDataOutputStream.sync()).
void	flush() Flushes the stream, writing any data currently buffered in stream implementation to the proper output stream.
long	getPos() Gets the position of the stream (non-negative), defined as the number of bytes from the beginning of the file to the current writing position.
static S3RecoverableFsDataOutputStream	newStream(org.apache.flink.fs.s3.common.writer.RecoverableMultiPartUpload upload, FunctionWithException<File,RefCountedFile,IOException> tmpFileCreator, long userDefinedMinPartSize)
RecoverableWriter.ResumeRecoverable	persist() Ensures all data so far is persistent (similar to FSDataOutputStream.sync()) and returns a handle to recover the stream at the current position.
static S3RecoverableFsDataOutputStream	recoverStream(org.apache.flink.fs.s3.common.writer.RecoverableMultiPartUpload upload, FunctionWithException<File,RefCountedFile,IOException> tmpFileCreator, long userDefinedMinPartSize, long bytesBeforeCurrentPart)
void	sync() Flushes the data all the way to the persistent non-volatile storage (for example disks).
void	write(byte[] b, int off, int len)
void	write(int b)

Methods inherited from class java.io.OutputStream

write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

write

public void write(int b)
           throws IOException

Specified by:

write in class OutputStream

Throws:

IOException

write

public void write(byte[] b,
                  int off,
                  int len)
           throws IOException

Overrides:

write in class OutputStream

Throws:

IOException

flush

public void flush()
           throws IOException

Description copied from class: FSDataOutputStream

Flushes the stream, writing any data currently buffered in stream implementation to the proper output stream. After this method has been called, the stream implementation must not hold onto any buffered data any more.

A completed flush does not mean that the data is necessarily persistent. Data persistence can is only assumed after calls to FSDataOutputStream.close() or FSDataOutputStream.sync().

Implementation note: This overrides the method defined in OutputStream as abstract to force implementations of the FSDataOutputStream to implement this method directly.

Specified by:

flush in interface Flushable

Specified by:

flush in class FSDataOutputStream

Throws:

IOException - Thrown if an I/O error occurs while flushing the stream.

getPos

public long getPos()
            throws IOException

Description copied from class: FSDataOutputStream

Gets the position of the stream (non-negative), defined as the number of bytes from the beginning of the file to the current writing position. The position corresponds to the zero-based index of the next byte that will be written.

This method must report accurately report the current position of the stream. Various components of the high-availability and recovery logic rely on the accurate

Specified by:

getPos in class FSDataOutputStream

Returns:

The current position in the stream, defined as the number of bytes from the beginning of the file to the current writing position.

Throws:

IOException - Thrown if an I/O error occurs while obtaining the position from the stream implementation.

sync

public void sync()
          throws IOException

Description copied from class: FSDataOutputStream

Flushes the data all the way to the persistent non-volatile storage (for example disks). The method behaves similar to the fsync function, forcing all data to be persistent on the devices.

Specified by:

sync in class FSDataOutputStream

Throws:

IOException - Thrown if an I/O error occurs

close

public void close()
           throws IOException

Description copied from class: RecoverableFsDataOutputStream

Closes this stream. Closing the steam releases the local resources that the stream uses, but does NOT result in durability of previously written data. This method should be interpreted as a "close in order to dispose" or "close on failure".

In order to persist all previously written data, one needs to call the RecoverableFsDataOutputStream.closeForCommit() method and call RecoverableFsDataOutputStream.Committer.commit() on the retured committer object.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Specified by:

close in class RecoverableFsDataOutputStream

Throws:

IOException - Thrown if an error occurred during closing.

persist

public RecoverableWriter.ResumeRecoverable persist()
                                            throws IOException

Description copied from class: RecoverableFsDataOutputStream

Ensures all data so far is persistent (similar to FSDataOutputStream.sync()) and returns a handle to recover the stream at the current position.

Specified by:

persist in class RecoverableFsDataOutputStream

Throws:

IOException

closeForCommit

public RecoverableFsDataOutputStream.Committer closeForCommit()
                                                       throws IOException

Description copied from class: RecoverableFsDataOutputStream

Closes the stream, ensuring persistence of all data (similar to FSDataOutputStream.sync()). This returns a Committer that can be used to publish (make visible) the file that the stream was writing to.

Specified by:

closeForCommit in class RecoverableFsDataOutputStream

Throws:

IOException

newStream

public static S3RecoverableFsDataOutputStream newStream(org.apache.flink.fs.s3.common.writer.RecoverableMultiPartUpload upload,
                                                        FunctionWithException<File,RefCountedFile,IOException> tmpFileCreator,
                                                        long userDefinedMinPartSize)
                                                 throws IOException

Throws:

IOException

recoverStream

public static S3RecoverableFsDataOutputStream recoverStream(org.apache.flink.fs.s3.common.writer.RecoverableMultiPartUpload upload,
                                                            FunctionWithException<File,RefCountedFile,IOException> tmpFileCreator,
                                                            long userDefinedMinPartSize,
                                                            long bytesBeforeCurrentPart)
                                                     throws IOException

Throws:

IOException