org.apache.flink.streaming.api.functions.sink.filesystem

Class HadoopPathBasedBulkFormatBuilder<IN,BucketID,T extends HadoopPathBasedBulkFormatBuilder<IN,BucketID,T>>

java.lang.Object

org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.BucketsBuilder<IN,BucketID,T>

org.apache.flink.streaming.api.functions.sink.filesystem.HadoopPathBasedBulkFormatBuilder<IN,BucketID,T>

All Implemented Interfaces:

Serializable

public class HadoopPathBasedBulkFormatBuilder<IN,BucketID,T extends HadoopPathBasedBulkFormatBuilder<IN,BucketID,T>>
extends StreamingFileSink.BucketsBuilder<IN,BucketID,T>

Buckets builder to create buckets that use HadoopPathBasedPartFileWriter.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.BucketsBuilder

DEFAULT_BUCKET_CHECK_INTERVAL

Constructor Summary

Constructors

Constructor and Description
HadoopPathBasedBulkFormatBuilder(org.apache.hadoop.fs.Path basePath, HadoopPathBasedBulkWriter.Factory<IN> writerFactory, Configuration configuration, BucketAssigner<IN,BucketID> assigner)
HadoopPathBasedBulkFormatBuilder(org.apache.hadoop.fs.Path basePath, HadoopPathBasedBulkWriter.Factory<IN> writerFactory, HadoopFileCommitterFactory fileCommitterFactory, Configuration configuration, BucketAssigner<IN,BucketID> assigner, CheckpointRollingPolicy<IN,BucketID> policy, BucketFactory<IN,BucketID> bucketFactory, OutputFileConfig outputFileConfig)

Method Summary

Modifier and Type	Method and Description
Buckets<IN,BucketID>	createBuckets(int subtaskIndex)
BucketWriter<IN,BucketID>	createBucketWriter()
T	withBucketAssigner(BucketAssigner<IN,BucketID> assigner)
T	withBucketFactory(BucketFactory<IN,BucketID> factory)
T	withConfiguration(Configuration configuration)
T	withOutputFileConfig(OutputFileConfig outputFileConfig)
T	withRollingPolicy(CheckpointRollingPolicy<IN,BucketID> rollingPolicy)

Methods inherited from class org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.BucketsBuilder

self

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopPathBasedBulkFormatBuilder

public HadoopPathBasedBulkFormatBuilder(org.apache.hadoop.fs.Path basePath,
                                        HadoopPathBasedBulkWriter.Factory<IN> writerFactory,
                                        Configuration configuration,
                                        BucketAssigner<IN,BucketID> assigner)

HadoopPathBasedBulkFormatBuilder

public HadoopPathBasedBulkFormatBuilder(org.apache.hadoop.fs.Path basePath,
                                        HadoopPathBasedBulkWriter.Factory<IN> writerFactory,
                                        HadoopFileCommitterFactory fileCommitterFactory,
                                        Configuration configuration,
                                        BucketAssigner<IN,BucketID> assigner,
                                        CheckpointRollingPolicy<IN,BucketID> policy,
                                        BucketFactory<IN,BucketID> bucketFactory,
                                        OutputFileConfig outputFileConfig)

Method Detail

withBucketAssigner

public T withBucketAssigner(BucketAssigner<IN,BucketID> assigner)

withRollingPolicy

public T withRollingPolicy(CheckpointRollingPolicy<IN,BucketID> rollingPolicy)

withBucketFactory

public T withBucketFactory(BucketFactory<IN,BucketID> factory)

withOutputFileConfig

public T withOutputFileConfig(OutputFileConfig outputFileConfig)

withConfiguration

public T withConfiguration(Configuration configuration)

createBucketWriter

public BucketWriter<IN,BucketID> createBucketWriter()

Specified by:

createBucketWriter in class StreamingFileSink.BucketsBuilder<IN,BucketID,T extends HadoopPathBasedBulkFormatBuilder<IN,BucketID,T>>

createBuckets

public Buckets<IN,BucketID> createBuckets(int subtaskIndex)
                                   throws IOException

Specified by:

createBuckets in class StreamingFileSink.BucketsBuilder<IN,BucketID,T extends HadoopPathBasedBulkFormatBuilder<IN,BucketID,T>>

Throws:

IOException