RollingSink (flink 1.8-SNAPSHOT API)

java.lang.Object
- org.apache.flink.api.common.functions.AbstractRichFunction
- - org.apache.flink.streaming.api.functions.sink.RichSinkFunction<T>
  - - org.apache.flink.streaming.connectors.fs.RollingSink<T>

Type Parameters:

T - Type of the elements emitted by this sink

All Implemented Interfaces:

Serializable, Function, RichFunction, InputTypeConfigurable, CheckpointListener, CheckpointedFunction, SinkFunction<T>

Deprecated.
use BucketingSink instead.
```
@Deprecated
public class RollingSink<T>
extends RichSinkFunction<T>
implements InputTypeConfigurable, CheckpointedFunction, CheckpointListener
```
Sink that emits its input elements to rolling FileSystem files. This is integrated with the checkpointing mechanism to provide exactly once semantics.
When creating the sink a basePath must be specified. The base directory contains one directory for every bucket. The bucket directories themselves contain several part files. These contain the actual written data.
The sink uses a Bucketer to determine the name of bucket directories inside the base directory. Whenever the Bucketer returns a different directory name than it returned before the sink will close the current part files inside that bucket and start the new bucket directory. The default bucketer is a DateTimeBucketer with date format string ""yyyy-MM-dd--HH". You can specify a custom Bucketer using setBucketer(Bucketer). For example, use NonRollingBucketer if you don't want to have buckets but still write part files in a fault-tolerant way.
The filenames of the part files contain the part prefix, the parallel subtask index of the sink and a rolling counter, for example "part-1-17". Per default the part prefix is "part" but this can be configured using setPartPrefix(String). When a part file becomes bigger than the batch size the current part file is closed, the part counter is increased and a new part file is created. The batch size defaults to 384MB, this can be configured using setBatchSize(long).
Part files can be in one of three states: in-progress, pending or finished. The reason for this is how the sink works together with the checkpointing mechanism to provide exactly-once semantics and fault-tolerance. The part file that is currently being written to is in-progress. Once a part file is closed for writing it becomes pending. When a checkpoint is successful the currently pending files will be moved to finished. If a failure occurs the pending files will be deleted to reset state to the last checkpoint. The data in in-progress files will also have to be rolled back. If the FileSystem supports the truncate call this will be used to reset the file back to a previous state. If not, a special file with the same name as the part file and the suffix ".valid-length" will be written that contains the length up to which the file contains valid data. When reading the file it must be ensured that it is only read up to that point. The prefixes and suffixes for the different file states and valid-length files can be configured, for example with setPendingSuffix(String).
Note: If checkpointing is not enabled the pending files will never be moved to the finished state. In that case, the pending suffix/prefix can be set to "" to make the sink work in a non-fault-tolerant way but still provide output without prefixes and suffixes.
The part files are written using an instance of Writer. By default StringWriter is used, which writes the result of toString() for every element. Separated by newlines. You can configure the writer using setWriter(Writer). For example, SequenceFileWriter can be used to write Hadoop SequenceFiles.
Example:
```
     new RollingSink<Tuple2<IntWritable, Text>>(outPath)
         .setWriter(new SequenceFileWriter<IntWritable, Text>())
         .setBucketer(new DateTimeBucketer("yyyy-MM-dd--HHmm")
 
```
This will create a sink that writes to SequenceFiles and rolls every minute.
See Also:

DateTimeBucketer, StringWriter, SequenceFileWriter, Serialized Form

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`static class`	`RollingSink.BucketState` Deprecated. This is used for keeping track of the current in-progress files and files that we mark for moving from pending to final location after we get a checkpoint-complete notification.

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.sink.SinkFunction
SinkFunction.Context<T>

Constructor Summary

Constructors
Constructor and Description

RollingSink(String basePath)
Deprecated.

Creates a new RollingSink that writes files to the given base directory.

Constructors
Constructor and Description
`RollingSink(String basePath)` Deprecated. Creates a new `RollingSink` that writes files to the given base directory.

Method Summary

All Methods Instance Methods Concrete Methods Deprecated Methods
Modifier and Type	Method and Description
`void`	`close()` Deprecated. Tear-down method for the user code.
`RollingSink<T>`	`disableCleanupOnOpen()` Deprecated. This option is deprecated and remains only for backwards compatibility. We do not clean up lingering files anymore.
`void`	`initializeState(FunctionInitializationContext context)` Deprecated. This method is called when the parallel function instance is created during distributed execution.
`void`	`invoke(T value)` Deprecated.
`void`	`notifyCheckpointComplete(long checkpointId)` Deprecated. This method is called as a notification once a distributed checkpoint has been completed.
`void`	`open(Configuration parameters)` Deprecated. Initialization method for the function.
`RollingSink<T>`	`setAsyncTimeout(long timeout)` Deprecated. Sets the default timeout for asynchronous operations such as recoverLease and truncate.
`RollingSink<T>`	`setBatchSize(long batchSize)` Deprecated. Sets the maximum bucket size in bytes.
`RollingSink<T>`	`setBucketer(Bucketer bucketer)` Deprecated. Sets the `Bucketer` to use for determining the bucket files to write to.
`RollingSink<T>`	`setFSConfig(Configuration config)` Deprecated. Specify a custom `Configuration` that will be used when creating the `FileSystem` for writing.
`RollingSink<T>`	`setFSConfig(Configuration config)` Deprecated. Specify a custom `Configuration` that will be used when creating the `FileSystem` for writing.
`RollingSink<T>`	`setInProgressPrefix(String inProgressPrefix)` Deprecated. Sets the prefix of in-progress part files.
`RollingSink<T>`	`setInProgressSuffix(String inProgressSuffix)` Deprecated. Sets the suffix of in-progress part files.
`void`	`setInputType(TypeInformation<?> type, ExecutionConfig executionConfig)` Deprecated. Method that is called on an `OutputFormat` when it is passed to the DataSet's output method.
`RollingSink<T>`	`setPartPrefix(String partPrefix)` Deprecated. Sets the prefix of part files.
`RollingSink<T>`	`setPendingPrefix(String pendingPrefix)` Deprecated. Sets the prefix of pending part files.
`RollingSink<T>`	`setPendingSuffix(String pendingSuffix)` Deprecated. Sets the suffix of pending part files.
`RollingSink<T>`	`setValidLengthPrefix(String validLengthPrefix)` Deprecated. Sets the prefix of valid-length files.
`RollingSink<T>`	`setValidLengthSuffix(String validLengthSuffix)` Deprecated. Sets the suffix of valid-length files.
`RollingSink<T>`	`setWriter(Writer<T> writer)` Deprecated. Sets the `Writer` to be used for writing the incoming elements to bucket files.
`void`	`snapshotState(FunctionSnapshotContext context)` Deprecated. This method is called when a snapshot for a checkpoint is requested.

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction
getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.functions.sink.SinkFunction
invoke

- Constructor Detail
  - RollingSink
```
public RollingSink(String basePath)
```
    Deprecated.
    
    Creates a new RollingSink that writes files to the given base directory.
    This uses aDateTimeBucketer as bucketer and a StringWriter has writer. The maximum bucket size is set to 384 MB.
    
    Parameters:
    
    basePath - The directory to which to write the bucket files.
- Method Detail
  - setFSConfig
```
public RollingSink<T> setFSConfig(Configuration config)
```
    Deprecated.
    
    Specify a custom Configuration that will be used when creating the FileSystem for writing.
  - setFSConfig
```
public RollingSink<T> setFSConfig(Configuration config)
```
    Deprecated.
    
    Specify a custom Configuration that will be used when creating the FileSystem for writing.
  - setInputType
```
public void setInputType(TypeInformation<?> type,
                         ExecutionConfig executionConfig)
```
    Deprecated.
    
    Description copied from interface: InputTypeConfigurable
    
    Method that is called on an OutputFormat when it is passed to the DataSet's output method. May be used to configures the output format based on the data type.
    
    Specified by:
    
    setInputType in interface InputTypeConfigurable
    
    Parameters:
    
    type - The data type of the input.
    
    executionConfig - The execution config for this parallel execution.
  - initializeState
```
public void initializeState(FunctionInitializationContext context)
                     throws Exception
```
    Deprecated.
    
    Description copied from interface: CheckpointedFunction
    
    This method is called when the parallel function instance is created during distributed execution. Functions typically set up their state storing data structures in this method.
    
    Specified by:
    
    initializeState in interface CheckpointedFunction
    
    Parameters:
    
    context - the context for initializing the operator
    
    Throws:
    
    Exception
  - open
```
public void open(Configuration parameters)
          throws Exception
```
    Deprecated.
    
    Description copied from interface: RichFunction
    Initialization method for the function. It is called before the actual working methods (like map or join) and thus suitable for one time setup work. For functions that are part of an iteration, this method will be invoked at the beginning of each iteration superstep.
    The configuration object passed to the function can be used for configuration and initialization. The configuration contains all parameters that were configured on the function in the program composition.
```
 public class MyMapper extends FilterFunction<String> {

     private String searchString;

     public void open(Configuration parameters) {
         this.searchString = parameters.getString("foo");
     }

     public boolean filter(String value) {
         return value.equals(searchString);
     }
 }
 
```
    By default, this method does nothing.
    Specified by:
    
    open in interface RichFunction
    
    Overrides:
    
    open in class AbstractRichFunction
    
    Parameters:
    
    parameters - The configuration containing the parameters attached to the contract.
    
    Throws:
    
    Exception - Implementations may forward exceptions, which are caught by the runtime. When the runtime catches an exception, it aborts the task and lets the fail-over logic decide whether to retry the task execution.
    
    See Also:
    
    Configuration
  - close
```
public void close()
           throws Exception
```
    Deprecated.
    
    Description copied from interface: RichFunction
    
    Tear-down method for the user code. It is called after the last call to the main working methods (e.g. map or join). For functions that are part of an iteration, this method will be invoked after each iteration superstep.
    This method can be used for clean up work.
    
    Specified by:
    
    close in interface RichFunction
    
    Overrides:
    
    close in class AbstractRichFunction
    
    Throws:
    
    Exception - Implementations may forward exceptions, which are caught by the runtime. When the runtime catches an exception, it aborts the task and lets the fail-over logic decide whether to retry the task execution.
  - invoke
```
public void invoke(T value)
            throws Exception
```
    Deprecated.
    
    Specified by:
    
    invoke in interface SinkFunction<T>
    
    Throws:
    
    Exception
  - notifyCheckpointComplete
```
public void notifyCheckpointComplete(long checkpointId)
                              throws Exception
```
    Deprecated.
    
    Description copied from interface: CheckpointListener
    
    This method is called as a notification once a distributed checkpoint has been completed. Note that any exception during this method will not cause the checkpoint to fail any more.
    
    Specified by:
    
    notifyCheckpointComplete in interface CheckpointListener
    
    Parameters:
    
    checkpointId - The ID of the checkpoint that has been completed.
    
    Throws:
    
    Exception
  - snapshotState
```
public void snapshotState(FunctionSnapshotContext context)
                   throws Exception
```
    Deprecated.
    
    Description copied from interface: CheckpointedFunction
    
    This method is called when a snapshot for a checkpoint is requested. This acts as a hook to the function to ensure that all state is exposed by means previously offered through FunctionInitializationContext when the Function was initialized, or offered now by FunctionSnapshotContext itself.
    
    Specified by:
    
    snapshotState in interface CheckpointedFunction
    
    Parameters:
    
    context - the context for drawing a snapshot of the operator
    
    Throws:
    
    Exception
  - setBatchSize
```
public RollingSink<T> setBatchSize(long batchSize)
```
    Deprecated.
    
    Sets the maximum bucket size in bytes.
    When a bucket part file becomes larger than this size a new bucket part file is started and the old one is closed. The name of the bucket files depends on the Bucketer.
    
    Parameters:
    
    batchSize - The bucket part file size in bytes.
  - setBucketer
```
public RollingSink<T> setBucketer(Bucketer bucketer)
```
    Deprecated.
    
    Sets the Bucketer to use for determining the bucket files to write to.
    
    Parameters:
    
    bucketer - The bucketer to use.
  - setWriter
```
public RollingSink<T> setWriter(Writer<T> writer)
```
    Deprecated.
    
    Sets the Writer to be used for writing the incoming elements to bucket files.
    
    Parameters:
    
    writer - The Writer to use.
  - setInProgressSuffix
```
public RollingSink<T> setInProgressSuffix(String inProgressSuffix)
```
    Deprecated.
    
    Sets the suffix of in-progress part files. The default is "in-progress".
  - setInProgressPrefix
```
public RollingSink<T> setInProgressPrefix(String inProgressPrefix)
```
    Deprecated.
    
    Sets the prefix of in-progress part files. The default is "_".
  - setPendingSuffix
```
public RollingSink<T> setPendingSuffix(String pendingSuffix)
```
    Deprecated.
    
    Sets the suffix of pending part files. The default is ".pending".
  - setPendingPrefix
```
public RollingSink<T> setPendingPrefix(String pendingPrefix)
```
    Deprecated.
    
    Sets the prefix of pending part files. The default is "_".
  - setValidLengthSuffix
```
public RollingSink<T> setValidLengthSuffix(String validLengthSuffix)
```
    Deprecated.
    
    Sets the suffix of valid-length files. The default is ".valid-length".
  - setValidLengthPrefix
```
public RollingSink<T> setValidLengthPrefix(String validLengthPrefix)
```
    Deprecated.
    
    Sets the prefix of valid-length files. The default is "_".
  - setPartPrefix
```
public RollingSink<T> setPartPrefix(String partPrefix)
```
    Deprecated.
    
    Sets the prefix of part files. The default is "part".
  - disableCleanupOnOpen
```
@Deprecated
public RollingSink<T> disableCleanupOnOpen()
```
    Deprecated. This option is deprecated and remains only for backwards compatibility. We do not clean up lingering files anymore.
    
    Disable cleanup of leftover in-progress/pending files when the sink is opened.
    This should only be disabled if using the sink without checkpoints, to not remove the files already in the directory.
  - setAsyncTimeout
```
public RollingSink<T> setAsyncTimeout(long timeout)
```
    Deprecated.
    
    Sets the default timeout for asynchronous operations such as recoverLease and truncate.
    
    Parameters:
    
    timeout - The timeout, in milliseconds.

Back to Flink Website

Class RollingSink<T>

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.sink.SinkFunction

Constructor Summary

Method Summary

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

Methods inherited from class java.lang.Object

Methods inherited from interface org.apache.flink.streaming.api.functions.sink.SinkFunction

Constructor Detail

RollingSink

Method Detail

setFSConfig

setFSConfig

setInputType

initializeState

open

close

invoke

notifyCheckpointComplete

snapshotState

setBatchSize

setBucketer

setWriter

setInProgressSuffix

setInProgressPrefix

setPendingSuffix

setPendingPrefix

setValidLengthSuffix

setValidLengthPrefix

setPartPrefix

disableCleanupOnOpen

setAsyncTimeout

Back to Flink Website