org.apache.flink.streaming.api.functions.sink

Class TwoPhaseCommitSinkFunction<IN,TXN,CONTEXT>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.sink.RichSinkFunction<IN>

org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction<IN,TXN,CONTEXT>

Type Parameters:

IN - Input type for SinkFunction.

TXN - Transaction to store all of the information required to handle a transaction.

CONTEXT - Context that will be shared across all invocations for the given TwoPhaseCommitSinkFunction instance. Context is created once

All Implemented Interfaces:

Serializable, Function, RichFunction, CheckpointListener, CheckpointedFunction, SinkFunction<IN>

Deprecated.

This interface will be removed in future versions. Use the new Sink interface instead.

@Deprecated
 @PublicEvolving
public abstract class TwoPhaseCommitSinkFunction<IN,TXN,CONTEXT>
extends RichSinkFunction<IN>
implements CheckpointedFunction, CheckpointListener

This is a recommended base class for all of the SinkFunction that intend to implement exactly-once semantic. It does that by implementing two phase commit algorithm on top of the CheckpointedFunction and CheckpointListener. User should provide custom TXN (transaction handle) and implement abstract methods handling this transaction handle.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TwoPhaseCommitSinkFunction.State<TXN,CONTEXT> Deprecated. State POJO class coupling pendingTransaction, context and pendingCommitTransactions.
static class	TwoPhaseCommitSinkFunction.StateSerializer<TXN,CONTEXT> Deprecated. Custom TypeSerializer for the sink state.
static class	TwoPhaseCommitSinkFunction.StateSerializerSnapshot<TXN,CONTEXT> Deprecated. Snapshot for the TwoPhaseCommitSinkFunction.StateSerializer.
static class	TwoPhaseCommitSinkFunction.TransactionHolder<TXN> Deprecated. Adds metadata (currently only the start time of the transaction) to the transaction object.

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.sink.SinkFunction

SinkFunction.Context

Field Summary

Fields

Modifier and Type	Field and Description
protected LinkedHashMap<Long,TwoPhaseCommitSinkFunction.TransactionHolder<TXN>>	pendingCommitTransactions Deprecated.
protected ListState<TwoPhaseCommitSinkFunction.State<TXN,CONTEXT>>	state Deprecated.
protected Optional<CONTEXT>	userContext Deprecated.

Constructor Summary

Constructors

Constructor and Description
TwoPhaseCommitSinkFunction(TypeSerializer<TXN> transactionSerializer, TypeSerializer<CONTEXT> contextSerializer) Deprecated. Use default ListStateDescriptor for internal state serialization.

Method Summary

Modifier and Type	Method and Description
protected abstract void	abort(TXN transaction) Deprecated. Abort a transaction.
protected abstract TXN	beginTransaction() Deprecated. Method that starts a new transaction.
void	close() Deprecated. Tear-down method for the user code.
protected abstract void	commit(TXN transaction) Deprecated. Commit a pre-committed transaction.
protected TXN	currentTransaction() Deprecated.
protected TwoPhaseCommitSinkFunction<IN,TXN,CONTEXT>	enableTransactionTimeoutWarnings(double warningRatio) Deprecated. Enables logging of warnings if a transaction's elapsed time reaches a specified ratio of the transactionTimeout.
void	finish() Deprecated. This method is called at the end of data processing.
protected void	finishProcessing(TXN transaction) Deprecated. This method is called at the end of data processing.
protected void	finishRecoveringContext(Collection<TXN> handledTransactions) Deprecated. Callback for subclasses which is called after restoring (each) user context.
protected Optional<CONTEXT>	getUserContext() Deprecated.
protected TwoPhaseCommitSinkFunction<IN,TXN,CONTEXT>	ignoreFailuresAfterTransactionTimeout() Deprecated. If called, the sink will only log but not propagate exceptions thrown in recoverAndCommit(Object) if the transaction is older than a specified transaction timeout.
void	initializeState(FunctionInitializationContext context) Deprecated. This method is called when the parallel function instance is created during distributed execution.
protected Optional<CONTEXT>	initializeUserContext() Deprecated.
void	invoke(IN value) Deprecated. This should not be implemented by subclasses.
void	invoke(IN value, SinkFunction.Context context) Deprecated. Writes the given value to the sink.
protected abstract void	invoke(TXN transaction, IN value, SinkFunction.Context context) Deprecated. Write value within a transaction.
void	notifyCheckpointAborted(long checkpointId) Deprecated. This method is called as a notification once a distributed checkpoint has been aborted.
void	notifyCheckpointComplete(long checkpointId) Deprecated. Notifies the listener that the checkpoint with the given checkpointId completed and was committed.
protected Stream<Map.Entry<Long,TXN>>	pendingTransactions() Deprecated.
protected abstract void	preCommit(TXN transaction) Deprecated. Pre commit previously created transaction.
protected void	recoverAndAbort(TXN transaction) Deprecated. Abort a transaction that was rejected by a coordinator after a failure.
protected void	recoverAndCommit(TXN transaction) Deprecated. Invoked on recovered transactions after a failure.
protected TwoPhaseCommitSinkFunction<IN,TXN,CONTEXT>	setTransactionTimeout(long transactionTimeout) Deprecated. Sets the transaction timeout.
void	snapshotState(FunctionSnapshotContext context) Deprecated. This method is called when a snapshot for a checkpoint is requested.

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, open, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.functions.sink.SinkFunction

writeWatermark

Methods inherited from interface org.apache.flink.api.common.functions.RichFunction

open

Field Detail

pendingCommitTransactions

protected final LinkedHashMap<Long,TwoPhaseCommitSinkFunction.TransactionHolder<TXN>> pendingCommitTransactions

Deprecated.

userContext

protected transient Optional<CONTEXT> userContext

Deprecated.

state

protected transient ListState<TwoPhaseCommitSinkFunction.State<TXN,CONTEXT>> state

Deprecated.

Constructor Detail

TwoPhaseCommitSinkFunction

public TwoPhaseCommitSinkFunction(TypeSerializer<TXN> transactionSerializer,
                                  TypeSerializer<CONTEXT> contextSerializer)

Deprecated.

Use default ListStateDescriptor for internal state serialization. Helpful utilities for using this constructor are TypeInformation.of(Class), TypeHint and TypeInformation.of(TypeHint). Example:

 TwoPhaseCommitSinkFunction(TypeInformation.of(new TypeHint<State<TXN, CONTEXT>>() {}));
 

Parameters:

transactionSerializer - TypeSerializer for the transaction type of this sink

contextSerializer - TypeSerializer for the context type of this sink

Method Detail

initializeUserContext

protected Optional<CONTEXT> initializeUserContext()

Deprecated.

getUserContext

protected Optional<CONTEXT> getUserContext()

Deprecated.

currentTransaction

@Nullable
protected TXN currentTransaction()

Deprecated.

pendingTransactions

@Nonnull
protected Stream<Map.Entry<Long,TXN>> pendingTransactions()

Deprecated.

invoke

protected abstract void invoke(TXN transaction,
                               IN value,
                               SinkFunction.Context context)
                        throws Exception

Deprecated.

Write value within a transaction.

Throws:

Exception

beginTransaction

protected abstract TXN beginTransaction()
                                 throws Exception

Deprecated.

Method that starts a new transaction.

Returns:

newly created transaction.

Throws:

Exception

preCommit

protected abstract void preCommit(TXN transaction)
                           throws Exception

Deprecated.

Pre commit previously created transaction. Pre commit must make all of the necessary steps to prepare the transaction for a commit that might happen in the future. After this point the transaction might still be aborted, but underlying implementation must ensure that commit calls on already pre committed transactions will always succeed.

Usually implementation involves flushing the data.

Throws:

Exception

commit

protected abstract void commit(TXN transaction)

Deprecated.

Commit a pre-committed transaction. If this method fail, Flink application will be restarted and recoverAndCommit(Object) will be called again for the same transaction.

recoverAndCommit

protected void recoverAndCommit(TXN transaction)

Deprecated.

Invoked on recovered transactions after a failure. User implementation must ensure that this call will eventually succeed. If it fails, Flink application will be restarted and it will be invoked again. If it does not succeed eventually, a data loss will occur. Transactions will be recovered in an order in which they were created.

abort

protected abstract void abort(TXN transaction)

Deprecated.

Abort a transaction.

recoverAndAbort

protected void recoverAndAbort(TXN transaction)

Deprecated.

Abort a transaction that was rejected by a coordinator after a failure.

finishRecoveringContext

protected void finishRecoveringContext(Collection<TXN> handledTransactions)

Deprecated.

Callback for subclasses which is called after restoring (each) user context.

Parameters:

handledTransactions - transactions which were already committed or aborted and do not need further handling

finishProcessing

protected void finishProcessing(@Nullable
                                TXN transaction)

Deprecated.

This method is called at the end of data processing.

The method is expected to flush all remaining buffered data. Exceptions will cause the pipeline to be recognized as failed, because the last data items are not processed properly. You may use this method to flush remaining buffered elements in the state into the current transaction which will be committed in the last checkpoint.

invoke

public final void invoke(IN value)
                  throws Exception

Deprecated.

This should not be implemented by subclasses.

Specified by:

invoke in interface SinkFunction<IN>

Throws:

Exception

invoke

public final void invoke(IN value,
                         SinkFunction.Context context)
                  throws Exception

Deprecated.

Description copied from interface: SinkFunction

Writes the given value to the sink. This function is called for every record.

You have to override this method when implementing a SinkFunction, this is a default method for backward compatibility with the old-style method only.

Specified by:

invoke in interface SinkFunction<IN>

Parameters:

value - The input record.

context - Additional context about the input record.

Throws:

Exception - This method may throw exceptions. Throwing an exception will cause the operation to fail and may trigger recovery.

finish

public final void finish()
                  throws Exception

Deprecated.

Description copied from interface: SinkFunction

This method is called at the end of data processing.

The method is expected to flush all remaining buffered data. Exceptions will cause the pipeline to be recognized as failed, because the last data items are not processed properly. You may use this method to flush remaining buffered elements in the state into transactions which you can commit in the last checkpoint.

NOTE:This method does not need to close any resources. You should release external resources in the AbstractRichFunction.close() method.

Specified by:

finish in interface SinkFunction<IN>

Throws:

Exception - This method may throw exceptions. Throwing an exception will cause the operation to fail and may trigger recovery.

notifyCheckpointComplete

public final void notifyCheckpointComplete(long checkpointId)
                                    throws Exception

Deprecated.

Description copied from interface: CheckpointListener

Notifies the listener that the checkpoint with the given checkpointId completed and was committed.

These notifications are "best effort", meaning they can sometimes be skipped. To behave properly, implementers need to follow the "Checkpoint Subsuming Contract". Please see the class-level JavaDocs for details.

Please note that checkpoints may generally overlap, so you cannot assume that the notifyCheckpointComplete() call is always for the latest prior checkpoint (or snapshot) that was taken on the function/operator implementing this interface. It might be for a checkpoint that was triggered earlier. Implementing the "Checkpoint Subsuming Contract" (see above) properly handles this situation correctly as well.

Please note that throwing exceptions from this method will not cause the completed checkpoint to be revoked. Throwing exceptions will typically cause task/job failure and trigger recovery.

Specified by:

notifyCheckpointComplete in interface CheckpointListener

Parameters:

checkpointId - The ID of the checkpoint that has been completed.

Throws:

Exception - This method can propagate exceptions, which leads to a failure/recovery for the task. Note that this will NOT lead to the checkpoint being revoked.

notifyCheckpointAborted

public void notifyCheckpointAborted(long checkpointId)

Deprecated.

Description copied from interface: CheckpointListener

This method is called as a notification once a distributed checkpoint has been aborted.

Important: The fact that a checkpoint has been aborted does NOT mean that the data and artifacts produced between the previous checkpoint and the aborted checkpoint are to be discarded. The expected behavior is as if this checkpoint was never triggered in the first place, and the next successful checkpoint simply covers a longer time span. See the "Checkpoint Subsuming Contract" in the class-level JavaDocs for details.

These notifications are "best effort", meaning they can sometimes be skipped.

This method is very rarely necessary to implement. The "best effort" guarantee, together with the fact that this method should not result in discarding any data (per the "Checkpoint Subsuming Contract") means it is mainly useful for earlier cleanups of auxiliary resources. One example is to pro-actively clear a local per-checkpoint state cache upon checkpoint failure.

Specified by:

notifyCheckpointAborted in interface CheckpointListener

Parameters:

checkpointId - The ID of the checkpoint that has been aborted.

snapshotState

public void snapshotState(FunctionSnapshotContext context)
                   throws Exception

Deprecated.

Description copied from interface: CheckpointedFunction

This method is called when a snapshot for a checkpoint is requested. This acts as a hook to the function to ensure that all state is exposed by means previously offered through FunctionInitializationContext when the Function was initialized, or offered now by FunctionSnapshotContext itself.

Specified by:

snapshotState in interface CheckpointedFunction

Parameters:

context - the context for drawing a snapshot of the operator

Throws:

Exception - Thrown, if state could not be created ot restored.

initializeState

public void initializeState(FunctionInitializationContext context)
                     throws Exception

Deprecated.

Description copied from interface: CheckpointedFunction

This method is called when the parallel function instance is created during distributed execution. Functions typically set up their state storing data structures in this method.

Specified by:

initializeState in interface CheckpointedFunction

Parameters:

context - the context for initializing the operator

Throws:

Exception - Thrown, if state could not be created ot restored.

close

public void close()
           throws Exception

Deprecated.

Description copied from interface: RichFunction

Tear-down method for the user code. It is called after the last call to the main working methods (e.g. map or join). For functions that are part of an iteration, this method will be invoked after each iteration superstep.

This method can be used for clean up work.

Specified by:

close in interface RichFunction

Overrides:

close in class AbstractRichFunction

Throws:

Exception - Implementations may forward exceptions, which are caught by the runtime. When the runtime catches an exception, it aborts the task and lets the fail-over logic decide whether to retry the task execution.

setTransactionTimeout

protected TwoPhaseCommitSinkFunction<IN,TXN,CONTEXT> setTransactionTimeout(long transactionTimeout)

Deprecated.

Sets the transaction timeout. Setting only the transaction timeout has no effect in itself.

Parameters:

transactionTimeout - The transaction timeout in ms.

See Also:

ignoreFailuresAfterTransactionTimeout(), enableTransactionTimeoutWarnings(double)

ignoreFailuresAfterTransactionTimeout

protected TwoPhaseCommitSinkFunction<IN,TXN,CONTEXT> ignoreFailuresAfterTransactionTimeout()

Deprecated.

If called, the sink will only log but not propagate exceptions thrown in recoverAndCommit(Object) if the transaction is older than a specified transaction timeout. The start time of an transaction is determined by System.currentTimeMillis(). By default, failures are propagated.

enableTransactionTimeoutWarnings

protected TwoPhaseCommitSinkFunction<IN,TXN,CONTEXT> enableTransactionTimeoutWarnings(double warningRatio)

Deprecated.

Enables logging of warnings if a transaction's elapsed time reaches a specified ratio of the transactionTimeout. If warningRatio is 0, a warning will be always logged when committing the transaction.

Parameters:

warningRatio - A value in the range [0,1].

Returns: