org.apache.flink.streaming.connectors.kafka.shuffle

Class FlinkKafkaShuffleConsumer<T>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction<T>

org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase<T>

org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer<T>

org.apache.flink.streaming.connectors.kafka.shuffle.FlinkKafkaShuffleConsumer<T>

All Implemented Interfaces:

Serializable, Function, RichFunction, CheckpointListener, ResultTypeQueryable<T>, CheckpointedFunction, ParallelSourceFunction<T>, SourceFunction<T>

@Internal
public class FlinkKafkaShuffleConsumer<T>
extends FlinkKafkaConsumer<T>

Flink Kafka Shuffle Consumer Function.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

SourceFunction.SourceContext<T>

Field Summary

Fields inherited from class org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer

DEFAULT_POLL_TIMEOUT, KEY_POLL_TIMEOUT, pollTimeout, properties

Fields inherited from class org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase

deserializer, KEY_DISABLE_METRICS, KEY_PARTITION_DISCOVERY_INTERVAL_MILLIS, LOG, MAX_NUM_PENDING_CHECKPOINTS, PARTITION_DISCOVERY_DISABLED

Method Summary

Modifier and Type

Method and Description

protected AbstractFetcher<T,?>

createFetcher(SourceFunction.SourceContext<T> sourceContext, Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets, SerializedValue<WatermarkStrategy<T>> watermarkStrategy, StreamingRuntimeContext runtimeContext, OffsetCommitMode offsetCommitMode, MetricGroup consumerMetricGroup, boolean useMetrics) Creates the fetcher that connect to the Kafka brokers, pulls data, deserialized the data, and emits it into the data streams.

Methods inherited from class org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer

createPartitionDiscoverer, fetchOffsetsWithTimestamp, getIsAutoCommitEnabled

Methods inherited from class org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase

adjustAutoCommitConfig, assignTimestampsAndWatermarks, assignTimestampsAndWatermarks, assignTimestampsAndWatermarks, cancel, close, disableFilterRestoredPartitionsWithSubscribedTopics, getEnableCommitOnCheckpoints, getProducedType, initializeState, notifyCheckpointAborted, notifyCheckpointComplete, open, run, setCommitOffsetsOnCheckpoints, setStartFromEarliest, setStartFromGroupOffsets, setStartFromLatest, setStartFromSpecificOffsets, setStartFromTimestamp, snapshotState

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

createFetcher

protected AbstractFetcher<T,?> createFetcher(SourceFunction.SourceContext<T> sourceContext,
                                             Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets,
                                             SerializedValue<WatermarkStrategy<T>> watermarkStrategy,
                                             StreamingRuntimeContext runtimeContext,
                                             OffsetCommitMode offsetCommitMode,
                                             MetricGroup consumerMetricGroup,
                                             boolean useMetrics)
                                      throws Exception

Description copied from class: FlinkKafkaConsumerBase

Creates the fetcher that connect to the Kafka brokers, pulls data, deserialized the data, and emits it into the data streams.

Overrides:

createFetcher in class FlinkKafkaConsumer<T>

Parameters:

sourceContext - The source context to emit data to.

assignedPartitionsWithInitialOffsets - The set of partitions that this subtask should handle, with their start offsets.

watermarkStrategy - Optional, a serialized WatermarkStrategy.

runtimeContext - The task's runtime context.

Returns:

The instantiated fetcher

Throws:

Exception - The method should forward exceptions