org.apache.flink.connector.kafka.source.reader.fetcher

Class KafkaSourceFetcherManager

java.lang.Object

org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager<E,SplitT>

org.apache.flink.connector.base.source.reader.fetcher.SingleThreadFetcherManager<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,KafkaPartitionSplit>

org.apache.flink.connector.kafka.source.reader.fetcher.KafkaSourceFetcherManager

@Internal
public class KafkaSourceFetcherManager
extends SingleThreadFetcherManager<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,KafkaPartitionSplit>

The SplitFetcherManager for Kafka source. This class is needed to help commit the offsets to Kafka using the KafkaConsumer inside the KafkaPartitionSplitReader.

Field Summary

Fields inherited from class org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager

fetchers

Constructor Summary

Constructors

Constructor and Description

KafkaSourceFetcherManager(FutureCompletingBlockingQueue<RecordsWithSplitIds<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>>> elementsQueue, java.util.function.Supplier<SplitReader<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,KafkaPartitionSplit>> splitReaderSupplier, java.util.function.Consumer<Collection<String>> splitFinishedHook) Creates a new SplitFetcherManager with a single I/O threads.

Method Summary

Modifier and Type	Method and Description
void	commitOffsets(Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> offsetsToCommit, org.apache.kafka.clients.consumer.OffsetCommitCallback callback)

Methods inherited from class org.apache.flink.connector.base.source.reader.fetcher.SingleThreadFetcherManager

addSplits, getRunningFetcher

Methods inherited from class org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager

checkErrors, close, createSplitFetcher, getNumAliveFetchers, maybeShutdownFinishedFetchers, startFetcher

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

KafkaSourceFetcherManager

public KafkaSourceFetcherManager(FutureCompletingBlockingQueue<RecordsWithSplitIds<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>>> elementsQueue,
                                 java.util.function.Supplier<SplitReader<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,KafkaPartitionSplit>> splitReaderSupplier,
                                 java.util.function.Consumer<Collection<String>> splitFinishedHook)

Creates a new SplitFetcherManager with a single I/O threads.

Parameters:

elementsQueue - The queue that is used to hand over data from the I/O thread (the fetchers) to the reader (which emits the records and book-keeps the state. This must be the same queue instance that is also passed to the SourceReaderBase.

splitReaderSupplier - The factory for the split reader that connects to the source system.

splitFinishedHook - Hook for handling finished splits in split fetchers.

Method Detail

commitOffsets

public void commitOffsets(Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> offsetsToCommit,
                          org.apache.kafka.clients.consumer.OffsetCommitCallback callback)