org.apache.flink.streaming.connectors.kafka.internal

Class KafkaShuffleFetcher<T>

java.lang.Object

org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher<T,org.apache.kafka.common.TopicPartition>

org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher<T>

org.apache.flink.streaming.connectors.kafka.internal.KafkaShuffleFetcher<T>

@Internal
public class KafkaShuffleFetcher<T>
extends KafkaFetcher<T>

Fetch data from Kafka for Kafka Shuffle.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	KafkaShuffleFetcher.KafkaShuffleElement An element in a KafkaShuffle.
static class	KafkaShuffleFetcher.KafkaShuffleElementDeserializer<T> Deserializer for KafkaShuffleElement.
static class	KafkaShuffleFetcher.KafkaShuffleRecord<T> One value with Type T in a KafkaShuffle.
static class	KafkaShuffleFetcher.KafkaShuffleWatermark A watermark element in a KafkaShuffle.

Field Summary

Fields inherited from class org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher

checkpointLock, sourceContext, unassignedPartitionsQueue, watermarkOutput

Constructor Summary

Constructors

Constructor and Description

KafkaShuffleFetcher(SourceFunction.SourceContext<T> sourceContext, Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets, SerializedValue<WatermarkStrategy<T>> watermarkStrategy, ProcessingTimeService processingTimeProvider, long autoWatermarkInterval, ClassLoader userCodeClassLoader, String taskNameWithSubtasks, KafkaDeserializationSchema<T> deserializer, Properties kafkaProperties, long pollTimeout, MetricGroup subtaskMetricGroup, MetricGroup consumerMetricGroup, boolean useMetrics, TypeSerializer<T> typeSerializer, int producerParallelism)

Method Summary

Modifier and Type	Method and Description
protected String	getFetcherName() Gets the name of this fetcher, for thread naming and logging purposes.
protected void	partitionConsumerRecordsHandler(List<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>> partitionRecords, KafkaTopicPartitionState<T,org.apache.kafka.common.TopicPartition> partition)

Methods inherited from class org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher

cancel, createKafkaPartitionHandle, doCommitInternalOffsetsToKafka, runFetchLoop

Methods inherited from class org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher

addDiscoveredPartitions, commitInternalOffsetsToKafka, emitRecordsWithTimestamps, snapshotCurrentState, subscribedPartitionStates

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

KafkaShuffleFetcher

public KafkaShuffleFetcher(SourceFunction.SourceContext<T> sourceContext,
                           Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets,
                           SerializedValue<WatermarkStrategy<T>> watermarkStrategy,
                           ProcessingTimeService processingTimeProvider,
                           long autoWatermarkInterval,
                           ClassLoader userCodeClassLoader,
                           String taskNameWithSubtasks,
                           KafkaDeserializationSchema<T> deserializer,
                           Properties kafkaProperties,
                           long pollTimeout,
                           MetricGroup subtaskMetricGroup,
                           MetricGroup consumerMetricGroup,
                           boolean useMetrics,
                           TypeSerializer<T> typeSerializer,
                           int producerParallelism)
                    throws Exception

Throws:

Exception

Method Detail

getFetcherName

protected String getFetcherName()

Description copied from class: KafkaFetcher

Gets the name of this fetcher, for thread naming and logging purposes.

Overrides:

getFetcherName in class KafkaFetcher<T>

partitionConsumerRecordsHandler

protected void partitionConsumerRecordsHandler(List<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>> partitionRecords,
                                               KafkaTopicPartitionState<T,org.apache.kafka.common.TopicPartition> partition)
                                        throws Exception

Overrides:

partitionConsumerRecordsHandler in class KafkaFetcher<T>

Throws:

Exception