org.apache.flink.connector.file.table.stream

Class StreamingSink

java.lang.Object

org.apache.flink.connector.file.table.stream.StreamingSink

@Internal
public class StreamingSink
extends Object

Helper for creating streaming file sink.

Method Summary

Modifier and Type	Method and Description
static <T> DataStream<PartitionCommitInfo>	compactionWriter(ProviderContext providerContext, DataStream<T> inputStream, long bucketCheckInterval, StreamingFileSink.BucketsBuilder<T,String,? extends StreamingFileSink.BucketsBuilder<T,String,?>> bucketsBuilder, FileSystemFactory fsFactory, Path path, CompactReader.Factory<T> readFactory, long targetFileSize, int parallelism, boolean parallelismConfigured) Create a file writer with compaction operators by input stream.
static DataStreamSink<?>	sink(ProviderContext providerContext, DataStream<PartitionCommitInfo> writer, Path locationPath, ObjectIdentifier identifier, List<String> partitionKeys, TableMetaStoreFactory msFactory, FileSystemFactory fsFactory, Configuration options) Create a sink from file writer.
static <T> DataStream<PartitionCommitInfo>	writer(ProviderContext providerContext, DataStream<T> inputStream, long bucketCheckInterval, StreamingFileSink.BucketsBuilder<T,String,? extends StreamingFileSink.BucketsBuilder<T,String,?>> bucketsBuilder, int parallelism, List<String> partitionKeys, Configuration conf, boolean parallelismConfigured) Create a file writer by input stream.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

writer

public static <T> DataStream<PartitionCommitInfo> writer(ProviderContext providerContext,
                                                         DataStream<T> inputStream,
                                                         long bucketCheckInterval,
                                                         StreamingFileSink.BucketsBuilder<T,String,? extends StreamingFileSink.BucketsBuilder<T,String,?>> bucketsBuilder,
                                                         int parallelism,
                                                         List<String> partitionKeys,
                                                         Configuration conf,
                                                         boolean parallelismConfigured)

Create a file writer by input stream. This is similar to StreamingFileSink, in addition, it can emit PartitionCommitInfo to down stream.

compactionWriter

public static <T> DataStream<PartitionCommitInfo> compactionWriter(ProviderContext providerContext,
                                                                   DataStream<T> inputStream,
                                                                   long bucketCheckInterval,
                                                                   StreamingFileSink.BucketsBuilder<T,String,? extends StreamingFileSink.BucketsBuilder<T,String,?>> bucketsBuilder,
                                                                   FileSystemFactory fsFactory,
                                                                   Path path,
                                                                   CompactReader.Factory<T> readFactory,
                                                                   long targetFileSize,
                                                                   int parallelism,
                                                                   boolean parallelismConfigured)

Create a file writer with compaction operators by input stream. In addition, it can emit PartitionCommitInfo to down stream.

sink

public static DataStreamSink<?> sink(ProviderContext providerContext,
                                     DataStream<PartitionCommitInfo> writer,
                                     Path locationPath,
                                     ObjectIdentifier identifier,
                                     List<String> partitionKeys,
                                     TableMetaStoreFactory msFactory,
                                     FileSystemFactory fsFactory,
                                     Configuration options)

Create a sink from file writer. Decide whether to add the node to commit partitions according to options.