org.apache.flink.table.descriptors

Class Kafka

java.lang.Object

org.apache.flink.table.descriptors.DescriptorBase

org.apache.flink.table.descriptors.ConnectorDescriptor

org.apache.flink.table.descriptors.Kafka

All Implemented Interfaces:

Descriptor

@PublicEvolving
public class Kafka
extends ConnectorDescriptor

Connector descriptor for the Apache Kafka message queue.

Constructor Summary

Constructors

Constructor and Description
Kafka() Connector descriptor for the Apache Kafka message queue.

Method Summary

Modifier and Type	Method and Description
Kafka	properties(Properties properties) Sets the configuration properties for the Kafka consumer.
Kafka	property(String key, String value) Adds a configuration properties for the Kafka consumer.
Kafka	sinkPartitionerCustom(Class<? extends FlinkKafkaPartitioner> partitionerClass) Configures how to partition records from Flink's partitions into Kafka's partitions.
Kafka	sinkPartitionerFixed() Configures how to partition records from Flink's partitions into Kafka's partitions.
Kafka	sinkPartitionerRoundRobin() Configures how to partition records from Flink's partitions into Kafka's partitions.
Kafka	startFromEarliest() Configures to start reading from the earliest offset for all partitions.
Kafka	startFromGroupOffsets() Configures to start reading from any committed group offsets found in Zookeeper / Kafka brokers.
Kafka	startFromLatest() Configures to start reading from the latest offset for all partitions.
Kafka	startFromSpecificOffset(int partition, long specificOffset) Configures to start reading partitions from specific offsets and specifies the given offset for the given partition.
Kafka	startFromSpecificOffsets(Map<Integer,Long> specificOffsets) Configures to start reading partitions from specific offsets, set independently for each partition.
Kafka	startFromTimestamp(long startTimestampMillis) Configures to start reading from partition offsets of the specified timestamp.
protected Map<String,String>	toConnectorProperties() Converts this descriptor into a set of connector properties.
Kafka	topic(String topic) Sets the topic from which the table is read.
Kafka	version(String version) Sets the Kafka version to be used.

Methods inherited from class org.apache.flink.table.descriptors.ConnectorDescriptor

isFormatNeeded, toProperties

Methods inherited from class org.apache.flink.table.descriptors.DescriptorBase

toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

Kafka

public Kafka()

Connector descriptor for the Apache Kafka message queue.

Method Detail

version

public Kafka version(String version)

Sets the Kafka version to be used.

Parameters:

version - Kafka version. E.g., "0.8", "0.11", etc.

topic

public Kafka topic(String topic)

Sets the topic from which the table is read.

Parameters:

topic - The topic from which the table is read.

properties

public Kafka properties(Properties properties)

Sets the configuration properties for the Kafka consumer. Resets previously set properties.

Parameters:

properties - The configuration properties for the Kafka consumer.

property

public Kafka property(String key,
                      String value)

Adds a configuration properties for the Kafka consumer.

Parameters:

key - property key for the Kafka consumer

value - property value for the Kafka consumer

startFromEarliest

public Kafka startFromEarliest()

Configures to start reading from the earliest offset for all partitions.

See Also:

FlinkKafkaConsumerBase.setStartFromEarliest()

startFromLatest

public Kafka startFromLatest()

Configures to start reading from the latest offset for all partitions.

See Also:

FlinkKafkaConsumerBase.setStartFromLatest()

startFromGroupOffsets

public Kafka startFromGroupOffsets()

Configures to start reading from any committed group offsets found in Zookeeper / Kafka brokers.

See Also:

FlinkKafkaConsumerBase.setStartFromGroupOffsets()

startFromSpecificOffsets

public Kafka startFromSpecificOffsets(Map<Integer,Long> specificOffsets)

Configures to start reading partitions from specific offsets, set independently for each partition. Resets previously set offsets.

Parameters:

specificOffsets - the specified offsets for partitions

See Also:

FlinkKafkaConsumerBase.setStartFromSpecificOffsets(Map)

startFromSpecificOffset

public Kafka startFromSpecificOffset(int partition,
                                     long specificOffset)

Configures to start reading partitions from specific offsets and specifies the given offset for the given partition.

Parameters:

partition - partition index

specificOffset - partition offset to start reading from

See Also:

FlinkKafkaConsumerBase.setStartFromSpecificOffsets(Map)

startFromTimestamp

public Kafka startFromTimestamp(long startTimestampMillis)

Configures to start reading from partition offsets of the specified timestamp.

Parameters:

startTimestampMillis - timestamp to start reading from

See Also:

FlinkKafkaConsumerBase.setStartFromTimestamp(long)

sinkPartitionerFixed

public Kafka sinkPartitionerFixed()

Configures how to partition records from Flink's partitions into Kafka's partitions.

This strategy ensures that each Flink partition ends up in one Kafka partition.

Note: One Kafka partition can contain multiple Flink partitions. Examples:

More Flink partitions than Kafka partitions. Some (or all) Kafka partitions contain the output of more than one flink partition:

     Flink Sinks            Kafka Partitions
         1    ---------------->    1
         2    --------------/
         3    -------------/
         4    ------------/
 

Fewer Flink partitions than Kafka partitions:

     Flink Sinks            Kafka Partitions
         1    ---------------->    1
         2    ---------------->    2
                                      3
                                      4
                                      5
 

See Also:

FlinkFixedPartitioner

sinkPartitionerRoundRobin

public Kafka sinkPartitionerRoundRobin()

Configures how to partition records from Flink's partitions into Kafka's partitions.

This strategy ensures that records will be distributed to Kafka partitions in a round-robin fashion.

Note: This strategy is useful to avoid an unbalanced partitioning. However, it will cause a lot of network connections between all the Flink instances and all the Kafka brokers.

sinkPartitionerCustom

public Kafka sinkPartitionerCustom(Class<? extends FlinkKafkaPartitioner> partitionerClass)

Configures how to partition records from Flink's partitions into Kafka's partitions.

This strategy allows for a custom partitioner by providing an implementation of FlinkKafkaPartitioner.

toConnectorProperties

protected Map<String,String> toConnectorProperties()

Description copied from class: ConnectorDescriptor

Converts this descriptor into a set of connector properties. Usually prefixed with ConnectorDescriptorValidator.CONNECTOR.

Specified by:

toConnectorProperties in class ConnectorDescriptor