org.apache.flink.streaming.connectors.kafka.internal

Class Kafka09Fetcher<T>

java.lang.Object

org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher<T,org.apache.kafka.common.TopicPartition>

org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher<T>

Type Parameters:

T - The type of elements produced by the fetcher.

Direct Known Subclasses:

Kafka010Fetcher

public class Kafka09Fetcher<T>
extends AbstractFetcher<T,org.apache.kafka.common.TopicPartition>

A fetcher that fetches data from Kafka brokers via the Kafka 0.9 consumer API.

Field Summary

Fields inherited from class org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher

checkpointLock, NO_TIMESTAMPS_WATERMARKS, PERIODIC_WATERMARKS, PUNCTUATED_WATERMARKS, sourceContext, timestampWatermarkMode, unassignedPartitionsQueue

Constructor Summary

Constructors

Constructor and Description

Kafka09Fetcher(SourceFunction.SourceContext<T> sourceContext, Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets, SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic, SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated, ProcessingTimeService processingTimeProvider, long autoWatermarkInterval, ClassLoader userCodeClassLoader, String taskNameWithSubtasks, KeyedDeserializationSchema<T> deserializer, Properties kafkaProperties, long pollTimeout, MetricGroup subtaskMetricGroup, MetricGroup consumerMetricGroup, boolean useMetrics)

Method Summary

Modifier and Type	Method and Description
void	cancel()
protected KafkaConsumerCallBridge	createCallBridge()
org.apache.kafka.common.TopicPartition	createKafkaPartitionHandle(KafkaTopicPartition partition) Creates the Kafka version specific representation of the given topic partition.
protected void	doCommitInternalOffsetsToKafka(Map<KafkaTopicPartition,Long> offsets, KafkaCommitCallback commitCallback)
protected void	emitRecord(T record, KafkaTopicPartitionState<org.apache.kafka.common.TopicPartition> partition, long offset, org.apache.kafka.clients.consumer.ConsumerRecord<?,?> consumerRecord)
protected String	getFetcherName() Gets the name of this fetcher, for thread naming and logging purposes.
void	runFetchLoop()

Methods inherited from class org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher

addDiscoveredPartitions, commitInternalOffsetsToKafka, emitRecord, emitRecordWithTimestamp, emitRecordWithTimestampAndPeriodicWatermark, emitRecordWithTimestampAndPunctuatedWatermark, snapshotCurrentState, subscribedPartitionStates

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Kafka09Fetcher

public Kafka09Fetcher(SourceFunction.SourceContext<T> sourceContext,
                      Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets,
                      SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,
                      SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,
                      ProcessingTimeService processingTimeProvider,
                      long autoWatermarkInterval,
                      ClassLoader userCodeClassLoader,
                      String taskNameWithSubtasks,
                      KeyedDeserializationSchema<T> deserializer,
                      Properties kafkaProperties,
                      long pollTimeout,
                      MetricGroup subtaskMetricGroup,
                      MetricGroup consumerMetricGroup,
                      boolean useMetrics)
               throws Exception

Throws:

Exception

Method Detail

runFetchLoop

public void runFetchLoop()
                  throws Exception

Specified by:

runFetchLoop in class AbstractFetcher<T,org.apache.kafka.common.TopicPartition>

Throws:

Exception

cancel

public void cancel()

Specified by:

cancel in class AbstractFetcher<T,org.apache.kafka.common.TopicPartition>

emitRecord

protected void emitRecord(T record,
                          KafkaTopicPartitionState<org.apache.kafka.common.TopicPartition> partition,
                          long offset,
                          org.apache.kafka.clients.consumer.ConsumerRecord<?,?> consumerRecord)
                   throws Exception

Throws:

Exception

getFetcherName

protected String getFetcherName()

Gets the name of this fetcher, for thread naming and logging purposes.

createCallBridge

protected KafkaConsumerCallBridge createCallBridge()

createKafkaPartitionHandle

public org.apache.kafka.common.TopicPartition createKafkaPartitionHandle(KafkaTopicPartition partition)

Description copied from class: AbstractFetcher

Creates the Kafka version specific representation of the given topic partition.

Specified by:

createKafkaPartitionHandle in class AbstractFetcher<T,org.apache.kafka.common.TopicPartition>

Parameters:

partition - The Flink representation of the Kafka topic partition.

Returns:

The version-specific Kafka representation of the Kafka topic partition.

doCommitInternalOffsetsToKafka

protected void doCommitInternalOffsetsToKafka(Map<KafkaTopicPartition,Long> offsets,
                                              @Nonnull
                                              KafkaCommitCallback commitCallback)
                                       throws Exception

Specified by:

doCommitInternalOffsetsToKafka in class AbstractFetcher<T,org.apache.kafka.common.TopicPartition>

Throws:

Exception