org.apache.flink.connector.file.table.batch.compact

Class BatchPartitionCommitterSink

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.sink.RichSinkFunction<CompactMessages.CompactOutput>

org.apache.flink.connector.file.table.batch.compact.BatchPartitionCommitterSink

All Implemented Interfaces:

Serializable, Function, RichFunction, SinkFunction<CompactMessages.CompactOutput>

public class BatchPartitionCommitterSink
extends RichSinkFunction<CompactMessages.CompactOutput>

Committer operator for partition in batch mode. This is the single (non-parallel) task. It collects all the partition information including partition and written files send from upstream.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.sink.SinkFunction

SinkFunction.Context

Constructor Summary

Constructors

Constructor and Description
BatchPartitionCommitterSink(FileSystemFactory fsFactory, TableMetaStoreFactory msFactory, boolean overwrite, boolean isToLocal, Path tmpPath, String[] partitionColumns, LinkedHashMap<String,String> staticPartitions, ObjectIdentifier identifier, PartitionCommitPolicyFactory partitionCommitPolicyFactory)

Method Summary

Modifier and Type	Method and Description
void	close() Tear-down method for the user code.
void	finish() This method is called at the end of data processing.
void	invoke(CompactMessages.CompactOutput compactOutput, SinkFunction.Context context) Writes the given value to the sink.
void	open(OpenContext openContext) Initialization method for the function.

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, open, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.functions.sink.SinkFunction

invoke, writeWatermark

Constructor Detail

BatchPartitionCommitterSink

public BatchPartitionCommitterSink(FileSystemFactory fsFactory,
                                   TableMetaStoreFactory msFactory,
                                   boolean overwrite,
                                   boolean isToLocal,
                                   Path tmpPath,
                                   String[] partitionColumns,
                                   LinkedHashMap<String,String> staticPartitions,
                                   ObjectIdentifier identifier,
                                   PartitionCommitPolicyFactory partitionCommitPolicyFactory)

Method Detail

open

public void open(OpenContext openContext)
          throws Exception

Description copied from interface: RichFunction

Initialization method for the function. It is called before the actual working methods (like map or join) and thus suitable for one time setup work. For functions that are part of an iteration, this method will be invoked at the beginning of each iteration superstep.

The openContext object passed to the function can be used for configuration and initialization. The openContext contains some necessary information that were configured on the function in the program composition.

 public class MyFilter extends RichFilterFunction<String> {

     private String searchString;

     public void open(OpenContext openContext) {
         // initialize the value of searchString
     }

     public boolean filter(String value) {
         return value.equals(searchString);
     }
 }
 

By default, this method does nothing.

1. If you implement open(OpenContext openContext), the open(OpenContext openContext) will be invoked and the open(Configuration parameters) won't be invoked. 2. If you don't implement open(OpenContext openContext), the open(Configuration parameters) will be invoked in the default implementation of the open(OpenContext openContext).

Parameters:

openContext - The context containing information about the context in which the function is opened.

Throws:

Exception - Implementations may forward exceptions, which are caught by the runtime. When the runtime catches an exception, it aborts the task and lets the fail-over logic decide whether to retry the task execution.

invoke

public void invoke(CompactMessages.CompactOutput compactOutput,
                   SinkFunction.Context context)
            throws Exception

Description copied from interface: SinkFunction

Writes the given value to the sink. This function is called for every record.

You have to override this method when implementing a SinkFunction, this is a default method for backward compatibility with the old-style method only.

Parameters:

compactOutput - The input record.

context - Additional context about the input record.

Throws:

Exception - This method may throw exceptions. Throwing an exception will cause the operation to fail and may trigger recovery.

finish

public void finish()
            throws Exception

Description copied from interface: SinkFunction

This method is called at the end of data processing.

The method is expected to flush all remaining buffered data. Exceptions will cause the pipeline to be recognized as failed, because the last data items are not processed properly. You may use this method to flush remaining buffered elements in the state into transactions which you can commit in the last checkpoint.

NOTE:This method does not need to close any resources. You should release external resources in the AbstractRichFunction.close() method.

Throws:

Exception - This method may throw exceptions. Throwing an exception will cause the operation to fail and may trigger recovery.

close

public void close()
           throws Exception

Description copied from interface: RichFunction

Tear-down method for the user code. It is called after the last call to the main working methods (e.g. map or join). For functions that are part of an iteration, this method will be invoked after each iteration superstep.

This method can be used for clean up work.

Specified by:

close in interface RichFunction

Overrides:

close in class AbstractRichFunction

Throws:

Exception - Implementations may forward exceptions, which are caught by the runtime. When the runtime catches an exception, it aborts the task and lets the fail-over logic decide whether to retry the task execution.