org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners

Class DateTimeBucketAssigner<IN>

java.lang.Object

org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.DateTimeBucketAssigner<IN>

All Implemented Interfaces:

Serializable, BucketAssigner<IN,String>

@PublicEvolving
public class DateTimeBucketAssigner<IN>
extends Object
implements BucketAssigner<IN,String>

A BucketAssigner that assigns to buckets based on current system time.

The DateTimeBucketer will create directories of the following form: /{basePath}/{dateTimePath}/. The basePath is the path that was specified as a base path when creating the new bucket. The dateTimePath is determined based on the current system time and the user provided format string.

DateTimeFormatter is used to derive a date string from the current system time and the date format string. The default format string is "yyyy-MM-dd--HH" so the rolling files will have a granularity of hours.

Example:

 BucketAssigner bucketAssigner = new DateTimeBucketAssigner("yyyy-MM-dd--HH");
 

This will create for example the following bucket path: /base/1976-12-31-14/

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.sink.filesystem.BucketAssigner

BucketAssigner.Context

Constructor Summary

Constructors

Constructor and Description
DateTimeBucketAssigner() Creates a new DateTimeBucketAssigner with format string "yyyy-MM-dd--HH".
DateTimeBucketAssigner(String formatString) Creates a new DateTimeBucketAssigner with the given date/time format string.
DateTimeBucketAssigner(String formatString, java.time.ZoneId zoneId) Creates a new DateTimeBucketAssigner with the given date/time format string using the given timezone.
DateTimeBucketAssigner(java.time.ZoneId zoneId) Creates a new DateTimeBucketAssigner with format string "yyyy-MM-dd--HH" using the given timezone.

Method Summary

Modifier and Type	Method and Description
String	getBucketId(IN element, BucketAssigner.Context context) Returns the identifier of the bucket the provided element should be put into.
SimpleVersionedSerializer<String>	getSerializer()
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

DateTimeBucketAssigner

public DateTimeBucketAssigner()

Creates a new DateTimeBucketAssigner with format string "yyyy-MM-dd--HH".

DateTimeBucketAssigner

public DateTimeBucketAssigner(String formatString)

Creates a new DateTimeBucketAssigner with the given date/time format string.

Parameters:

formatString - The format string that will be given to SimpleDateFormat to determine the bucket id.

DateTimeBucketAssigner

public DateTimeBucketAssigner(java.time.ZoneId zoneId)

Creates a new DateTimeBucketAssigner with format string "yyyy-MM-dd--HH" using the given timezone.

Parameters:

zoneId - The timezone used to format DateTimeFormatter for bucket id.

DateTimeBucketAssigner

public DateTimeBucketAssigner(String formatString,
                              java.time.ZoneId zoneId)

Creates a new DateTimeBucketAssigner with the given date/time format string using the given timezone.

Parameters:

formatString - The format string that will be given to DateTimeFormatter to determine the bucket path.

zoneId - The timezone used to format DateTimeFormatter for bucket id.

Method Detail

getBucketId

public String getBucketId(IN element,
                          BucketAssigner.Context context)

Description copied from interface: BucketAssigner

Returns the identifier of the bucket the provided element should be put into.

Specified by:

getBucketId in interface BucketAssigner<IN,String>

Parameters:

element - The current element being processed.

context - The context used by the current bucket assigner.

Returns:

A string representing the identifier of the bucket the element should be put into. The actual path to the bucket will result from the concatenation of the returned string and the base path provided during the initialization of the file sink.

getSerializer

public SimpleVersionedSerializer<String> getSerializer()

Specified by:

getSerializer in interface BucketAssigner<IN,String>

Returns:

A SimpleVersionedSerializer capable of serializing/deserializing the elements of type BucketID. That is the type of the objects returned by the BucketAssigner.getBucketId(Object, BucketAssigner.Context).

toString

public String toString()

Overrides:

toString in class Object