org.apache.flink.fs.gs.writer

Class GSRecoverableWriter

java.lang.Object

org.apache.flink.fs.gs.writer.GSRecoverableWriter

All Implemented Interfaces:

RecoverableWriter

public class GSRecoverableWriter
extends Object
implements RecoverableWriter

The recoverable writer implementation for Google storage.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.core.fs.RecoverableWriter

RecoverableWriter.CommitRecoverable, RecoverableWriter.ResumeRecoverable

Constructor Summary

Constructors

Constructor and Description
GSRecoverableWriter(GSBlobStorage storage, GSFileSystemOptions options) Construct a GS recoverable writer.

Method Summary

Modifier and Type	Method and Description
boolean	cleanupRecoverableState(RecoverableWriter.ResumeRecoverable resumable) Frees up any resources that were previously occupied in order to be able to recover from a (potential) failure.
SimpleVersionedSerializer<RecoverableWriter.CommitRecoverable>	getCommitRecoverableSerializer() The serializer for the CommitRecoverable types created in this writer.
SimpleVersionedSerializer<RecoverableWriter.ResumeRecoverable>	getResumeRecoverableSerializer() The serializer for the ResumeRecoverable types created in this writer.
RecoverableFsDataOutputStream	open(Path path) Opens a new recoverable stream to write to the given path.
RecoverableFsDataOutputStream	recover(RecoverableWriter.ResumeRecoverable resumable) Resumes a recoverable stream consistently at the point indicated by the given ResumeRecoverable.
RecoverableFsDataOutputStream.Committer	recoverForCommit(RecoverableWriter.CommitRecoverable resumable) Recovers a recoverable stream consistently at the point indicated by the given CommitRecoverable for finalizing and committing.
boolean	requiresCleanupOfRecoverableState() Marks if the writer requires to do any additional cleanup/freeing of resources occupied as part of a RecoverableWriter.ResumeRecoverable, e.g.
boolean	supportsResume() Checks whether the writer and its streams support resuming (appending to) files after recovery (via the RecoverableWriter.recover(ResumeRecoverable) method).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GSRecoverableWriter

public GSRecoverableWriter(GSBlobStorage storage,
                           GSFileSystemOptions options)

Construct a GS recoverable writer.

Parameters:

storage - The underlying blob storage instance

options - The GS file system options

Method Detail

requiresCleanupOfRecoverableState

public boolean requiresCleanupOfRecoverableState()

Description copied from interface: RecoverableWriter

Marks if the writer requires to do any additional cleanup/freeing of resources occupied as part of a RecoverableWriter.ResumeRecoverable, e.g. temporarily files created or objects uploaded to external systems.

In case cleanup is required, then RecoverableWriter.cleanupRecoverableState(ResumeRecoverable) should be called.

Specified by:

requiresCleanupOfRecoverableState in interface RecoverableWriter

Returns:

true if cleanup is required, false otherwise.

supportsResume

public boolean supportsResume()

Description copied from interface: RecoverableWriter

Checks whether the writer and its streams support resuming (appending to) files after recovery (via the RecoverableWriter.recover(ResumeRecoverable) method).

If true, then this writer supports the RecoverableWriter.recover(ResumeRecoverable) method. If false, then that method may not be supported and streams can only be recovered via RecoverableWriter.recoverForCommit(CommitRecoverable).

Specified by:

supportsResume in interface RecoverableWriter

open

public RecoverableFsDataOutputStream open(Path path)
                                   throws IOException

Description copied from interface: RecoverableWriter

Opens a new recoverable stream to write to the given path. Whether existing files will be overwritten is implementation specific and should not be relied upon.

Specified by:

open in interface RecoverableWriter

Parameters:

path - The path of the file/object to write to.

Returns:

A new RecoverableFsDataOutputStream writing a new file/object.

Throws:

IOException - Thrown if the stream could not be opened/initialized.

recover

public RecoverableFsDataOutputStream recover(RecoverableWriter.ResumeRecoverable resumable)

Description copied from interface: RecoverableWriter

Resumes a recoverable stream consistently at the point indicated by the given ResumeRecoverable. Future writes to the stream will continue / append the file as of that point.

This method is optional and whether it is supported is indicated through the RecoverableWriter.supportsResume() method.

Specified by:

recover in interface RecoverableWriter

Parameters:

resumable - The opaque handle with the recovery information.

Returns:

A recoverable stream writing to the file/object as it was at the point when the ResumeRecoverable was created.

cleanupRecoverableState

public boolean cleanupRecoverableState(RecoverableWriter.ResumeRecoverable resumable)

Description copied from interface: RecoverableWriter

Frees up any resources that were previously occupied in order to be able to recover from a (potential) failure. These can be temporary files that were written to the filesystem or objects that were uploaded to S3.

NOTE: This operation should not throw an exception, but return false if the cleanup did not happen for any reason.

Specified by:

cleanupRecoverableState in interface RecoverableWriter

Parameters:

resumable - The RecoverableWriter.ResumeRecoverable whose state we want to clean-up.

Returns:

true if the resources were successfully freed, false otherwise (e.g. the file to be deleted was not there for any reason - already deleted or never created).

recoverForCommit

public RecoverableFsDataOutputStream.Committer recoverForCommit(RecoverableWriter.CommitRecoverable resumable)

Description copied from interface: RecoverableWriter

Recovers a recoverable stream consistently at the point indicated by the given CommitRecoverable for finalizing and committing. This will publish the target file with exactly the data that was written up to the point then the CommitRecoverable was created.

Specified by:

recoverForCommit in interface RecoverableWriter

Parameters:

resumable - The opaque handle with the recovery information.

Returns:

A committer that publishes the target file.

getCommitRecoverableSerializer

public SimpleVersionedSerializer<RecoverableWriter.CommitRecoverable> getCommitRecoverableSerializer()

Description copied from interface: RecoverableWriter

The serializer for the CommitRecoverable types created in this writer. This serializer should be used to store the CommitRecoverable in checkpoint state or other forms of persistent state.

Specified by:

getCommitRecoverableSerializer in interface RecoverableWriter

getResumeRecoverableSerializer

public SimpleVersionedSerializer<RecoverableWriter.ResumeRecoverable> getResumeRecoverableSerializer()

Description copied from interface: RecoverableWriter

The serializer for the ResumeRecoverable types created in this writer. This serializer should be used to store the ResumeRecoverable in checkpoint state or other forms of persistent state.

Specified by:

getResumeRecoverableSerializer in interface RecoverableWriter