T
- The type of elements produced by the fetcher.public class Kafka09Fetcher<T> extends AbstractFetcher<T,org.apache.kafka.common.TopicPartition>
checkpointLock, NO_TIMESTAMPS_WATERMARKS, PERIODIC_WATERMARKS, PUNCTUATED_WATERMARKS, sourceContext, timestampWatermarkMode, useMetrics
Constructor and Description |
---|
Kafka09Fetcher(SourceFunction.SourceContext<T> sourceContext,
Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets,
SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,
SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,
ProcessingTimeService processingTimeProvider,
long autoWatermarkInterval,
ClassLoader userCodeClassLoader,
String taskNameWithSubtasks,
MetricGroup metricGroup,
KeyedDeserializationSchema<T> deserializer,
Properties kafkaProperties,
long pollTimeout,
boolean useMetrics) |
Modifier and Type | Method and Description |
---|---|
void |
cancel() |
void |
commitInternalOffsetsToKafka(Map<KafkaTopicPartition,Long> offsets,
KafkaCommitCallback commitCallback)
Commits the given partition offsets to the Kafka brokers (or to ZooKeeper for
older Kafka versions).
|
protected KafkaConsumerCallBridge |
createCallBridge() |
org.apache.kafka.common.TopicPartition |
createKafkaPartitionHandle(KafkaTopicPartition partition)
Creates the Kafka version specific representation of the given
topic partition.
|
protected void |
emitRecord(T record,
KafkaTopicPartitionState<org.apache.kafka.common.TopicPartition> partition,
long offset,
org.apache.kafka.clients.consumer.ConsumerRecord<?,?> consumerRecord) |
protected String |
getFetcherName()
Gets the name of this fetcher, for thread naming and logging purposes.
|
void |
runFetchLoop() |
addOffsetStateGauge, emitRecord, emitRecordWithTimestamp, emitRecordWithTimestampAndPeriodicWatermark, emitRecordWithTimestampAndPunctuatedWatermark, snapshotCurrentState, subscribedPartitionStates
public Kafka09Fetcher(SourceFunction.SourceContext<T> sourceContext, Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets, SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic, SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated, ProcessingTimeService processingTimeProvider, long autoWatermarkInterval, ClassLoader userCodeClassLoader, String taskNameWithSubtasks, MetricGroup metricGroup, KeyedDeserializationSchema<T> deserializer, Properties kafkaProperties, long pollTimeout, boolean useMetrics) throws Exception
Exception
public void runFetchLoop() throws Exception
runFetchLoop
in class AbstractFetcher<T,org.apache.kafka.common.TopicPartition>
Exception
public void cancel()
cancel
in class AbstractFetcher<T,org.apache.kafka.common.TopicPartition>
protected void emitRecord(T record, KafkaTopicPartitionState<org.apache.kafka.common.TopicPartition> partition, long offset, org.apache.kafka.clients.consumer.ConsumerRecord<?,?> consumerRecord) throws Exception
Exception
protected String getFetcherName()
protected KafkaConsumerCallBridge createCallBridge()
public org.apache.kafka.common.TopicPartition createKafkaPartitionHandle(KafkaTopicPartition partition)
AbstractFetcher
createKafkaPartitionHandle
in class AbstractFetcher<T,org.apache.kafka.common.TopicPartition>
partition
- The Flink representation of the Kafka topic partition.public void commitInternalOffsetsToKafka(Map<KafkaTopicPartition,Long> offsets, @Nonnull KafkaCommitCallback commitCallback) throws Exception
AbstractFetcher
OffsetCommitMode.ON_CHECKPOINTS
.
The given offsets are the internal checkpointed offsets, representing
the last processed record of each partition. Version-specific implementations of this method
need to hold the contract that the given offsets must be incremented by 1 before
committing them, so that committed offsets to Kafka represent "the next record to process".commitInternalOffsetsToKafka
in class AbstractFetcher<T,org.apache.kafka.common.TopicPartition>
offsets
- The offsets to commit to Kafka (implementations must increment offsets by 1 before committing).commitCallback
- The callback that the user should trigger when a commit request completes or fails.Exception
- This method forwards exceptions.Copyright © 2014–2018 The Apache Software Foundation. All rights reserved.