org.apache.flink.streaming.connectors.kafka.internals

Class Kafka08Fetcher<T>

java.lang.Object

org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher<T,kafka.common.TopicAndPartition>

org.apache.flink.streaming.connectors.kafka.internals.Kafka08Fetcher<T>

Type Parameters:

T - The type of elements produced by the fetcher.

public class Kafka08Fetcher<T>
extends AbstractFetcher<T,kafka.common.TopicAndPartition>

A fetcher that fetches data from Kafka brokers via the Kafka 0.8 low-level consumer API. The fetcher also handles the explicit communication with ZooKeeper to fetch initial offsets and to write offsets to ZooKeeper.

Field Summary

Fields inherited from class org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher

checkpointLock, NO_TIMESTAMPS_WATERMARKS, PERIODIC_WATERMARKS, PUNCTUATED_WATERMARKS, sourceContext, timestampWatermarkMode, unassignedPartitionsQueue

Constructor Summary

Constructors

Constructor and Description

Kafka08Fetcher(SourceFunction.SourceContext<T> sourceContext, Map<KafkaTopicPartition,Long> seedPartitionsWithInitialOffsets, SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic, SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated, StreamingRuntimeContext runtimeContext, KeyedDeserializationSchema<T> deserializer, Properties kafkaProperties, long autoCommitInterval, MetricGroup consumerMetricGroup, boolean useMetrics)

Method Summary

Modifier and Type	Method and Description
void	cancel()
protected kafka.common.TopicAndPartition	createKafkaPartitionHandle(KafkaTopicPartition partition) Creates the Kafka version specific representation of the given topic partition.
protected void	doCommitInternalOffsetsToKafka(Map<KafkaTopicPartition,Long> offsets, KafkaCommitCallback commitCallback)
void	runFetchLoop()

Methods inherited from class org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher

addDiscoveredPartitions, commitInternalOffsetsToKafka, emitRecord, emitRecordWithTimestamp, emitRecordWithTimestampAndPeriodicWatermark, emitRecordWithTimestampAndPunctuatedWatermark, snapshotCurrentState, subscribedPartitionStates

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Kafka08Fetcher

public Kafka08Fetcher(SourceFunction.SourceContext<T> sourceContext,
                      Map<KafkaTopicPartition,Long> seedPartitionsWithInitialOffsets,
                      SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,
                      SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,
                      StreamingRuntimeContext runtimeContext,
                      KeyedDeserializationSchema<T> deserializer,
                      Properties kafkaProperties,
                      long autoCommitInterval,
                      MetricGroup consumerMetricGroup,
                      boolean useMetrics)
               throws Exception

Throws:

Exception

Method Detail

runFetchLoop

public void runFetchLoop()
                  throws Exception

Specified by:

runFetchLoop in class AbstractFetcher<T,kafka.common.TopicAndPartition>

Throws:

Exception

cancel

public void cancel()

Specified by:

cancel in class AbstractFetcher<T,kafka.common.TopicAndPartition>

createKafkaPartitionHandle

protected kafka.common.TopicAndPartition createKafkaPartitionHandle(KafkaTopicPartition partition)

Description copied from class: AbstractFetcher

Creates the Kafka version specific representation of the given topic partition.

Specified by:

createKafkaPartitionHandle in class AbstractFetcher<T,kafka.common.TopicAndPartition>

Parameters:

partition - The Flink representation of the Kafka topic partition.

Returns:

The version-specific Kafka representation of the Kafka topic partition.

doCommitInternalOffsetsToKafka

protected void doCommitInternalOffsetsToKafka(Map<KafkaTopicPartition,Long> offsets,
                                              @Nonnull
                                              KafkaCommitCallback commitCallback)
                                       throws Exception

Specified by:

doCommitInternalOffsetsToKafka in class AbstractFetcher<T,kafka.common.TopicAndPartition>

Throws:

Exception