public class Kafka
extends org.apache.flink.table.descriptors.ConnectorDescriptor
Constructor and Description |
---|
Kafka()
Connector descriptor for the Apache Kafka message queue.
|
Modifier and Type | Method and Description |
---|---|
void |
addConnectorProperties(org.apache.flink.table.descriptors.DescriptorProperties properties)
Internal method for connector properties conversion.
|
Kafka |
properties(Properties properties)
Sets the configuration properties for the Kafka consumer.
|
Kafka |
property(String key,
String value)
Adds a configuration properties for the Kafka consumer.
|
Kafka |
sinkPartitionerCustom(Class<? extends FlinkKafkaPartitioner> partitionerClass)
Configures how to partition records from Flink's partitions into Kafka's partitions.
|
Kafka |
sinkPartitionerFixed()
Configures how to partition records from Flink's partitions into Kafka's partitions.
|
Kafka |
sinkPartitionerRoundRobin()
Configures how to partition records from Flink's partitions into Kafka's partitions.
|
Kafka |
startFromEarliest()
Configures to start reading from the earliest offset for all partitions.
|
Kafka |
startFromGroupOffsets()
Configures to start reading from any committed group offsets found in Zookeeper / Kafka brokers.
|
Kafka |
startFromLatest()
Configures to start reading from the latest offset for all partitions.
|
Kafka |
startFromSpecificOffset(int partition,
long specificOffset)
Configures to start reading partitions from specific offsets and specifies the given offset for
the given partition.
|
Kafka |
startFromSpecificOffsets(Map<Integer,Long> specificOffsets)
Configures to start reading partitions from specific offsets, set independently for each partition.
|
Kafka |
topic(String topic)
Sets the topic from which the table is read.
|
Kafka |
version(String version)
Sets the Kafka version to be used.
|
public Kafka version(String version)
version
- Kafka version. E.g., "0.8", "0.11", etc.public Kafka topic(String topic)
topic
- The topic from which the table is read.public Kafka properties(Properties properties)
properties
- The configuration properties for the Kafka consumer.public Kafka property(String key, String value)
key
- property key for the Kafka consumervalue
- property value for the Kafka consumerpublic Kafka startFromEarliest()
public Kafka startFromLatest()
public Kafka startFromGroupOffsets()
public Kafka startFromSpecificOffsets(Map<Integer,Long> specificOffsets)
specificOffsets
- the specified offsets for partitionsFlinkKafkaConsumerBase.setStartFromSpecificOffsets(Map)
public Kafka startFromSpecificOffset(int partition, long specificOffset)
partition
- partition indexspecificOffset
- partition offset to start reading fromFlinkKafkaConsumerBase.setStartFromSpecificOffsets(Map)
public Kafka sinkPartitionerFixed()
This strategy ensures that each Flink partition ends up in one Kafka partition.
Note: One Kafka partition can contain multiple Flink partitions. Examples:
More Flink partitions than Kafka partitions. Some (or all) Kafka partitions contain the output of more than one flink partition:
Flink Sinks Kafka Partitions 1 ----------------> 1 2 --------------/ 3 -------------/ 4 ------------/
Fewer Flink partitions than Kafka partitions:
Flink Sinks Kafka Partitions 1 ----------------> 1 2 ----------------> 2 3 4 5
FlinkFixedPartitioner
public Kafka sinkPartitionerRoundRobin()
This strategy ensures that records will be distributed to Kafka partitions in a round-robin fashion.
Note: This strategy is useful to avoid an unbalanced partitioning. However, it will cause a lot of network connections between all the Flink instances and all the Kafka brokers.
public Kafka sinkPartitionerCustom(Class<? extends FlinkKafkaPartitioner> partitionerClass)
This strategy allows for a custom partitioner by providing an implementation
of FlinkKafkaPartitioner
.
public void addConnectorProperties(org.apache.flink.table.descriptors.DescriptorProperties properties)
addConnectorProperties
in class org.apache.flink.table.descriptors.ConnectorDescriptor
Copyright © 2014–2020 The Apache Software Foundation. All rights reserved.