org.apache.flink.connector.base.source.reader.fetcher

Class SingleThreadFetcherManager<E,SplitT extends SourceSplit>

java.lang.Object

org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager<E,SplitT>

org.apache.flink.connector.base.source.reader.fetcher.SingleThreadFetcherManager<E,SplitT>

Direct Known Subclasses:

KafkaSourceFetcherManager

@Internal
public class SingleThreadFetcherManager<E,SplitT extends SourceSplit>
extends SplitFetcherManager<E,SplitT>

A Fetcher Manager with a single fetching thread (I/O thread) that handles all splits concurrently.

This pattern is, for example, useful for connectors like File Readers, Apache Kafka Readers, etc. In the example of Kafka, there is a single thread that reads all splits (topic partitions) via the same client. In the example of the file source, there is a single thread that reads the files after another.

Field Summary

Fields inherited from class org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager

fetchers

Constructor Summary

Constructors

Constructor and Description
SingleThreadFetcherManager(FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> elementsQueue, java.util.function.Supplier<SplitReader<E,SplitT>> splitReaderSupplier) Deprecated. Please use SingleThreadFetcherManager(FutureCompletingBlockingQueue, Supplier, Configuration) instead.
SingleThreadFetcherManager(FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> elementsQueue, java.util.function.Supplier<SplitReader<E,SplitT>> splitReaderSupplier, Configuration configuration) Creates a new SplitFetcherManager with a single I/O threads.
SingleThreadFetcherManager(FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> elementsQueue, java.util.function.Supplier<SplitReader<E,SplitT>> splitReaderSupplier, Configuration configuration, java.util.function.Consumer<Collection<String>> splitFinishedHook) Creates a new SplitFetcherManager with a single I/O threads.

Method Summary

Modifier and Type	Method and Description
void	addSplits(List<SplitT> splitsToAdd)
protected SplitFetcher<E,SplitT>	getRunningFetcher()

Methods inherited from class org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager

checkErrors, close, createSplitFetcher, getNumAliveFetchers, maybeShutdownFinishedFetchers, pauseOrResumeSplits, startFetcher

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SingleThreadFetcherManager

@Deprecated
public SingleThreadFetcherManager(FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> elementsQueue,
                                              java.util.function.Supplier<SplitReader<E,SplitT>> splitReaderSupplier)

Deprecated. Please use SingleThreadFetcherManager(FutureCompletingBlockingQueue, Supplier, Configuration) instead.

Creates a new SplitFetcherManager with a single I/O threads.

Parameters:

elementsQueue - The queue that is used to hand over data from the I/O thread (the fetchers) to the reader (which emits the records and book-keeps the state. This must be the same queue instance that is also passed to the SourceReaderBase.

splitReaderSupplier - The factory for the split reader that connects to the source system.

SingleThreadFetcherManager

public SingleThreadFetcherManager(FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> elementsQueue,
                                  java.util.function.Supplier<SplitReader<E,SplitT>> splitReaderSupplier,
                                  Configuration configuration)

Creates a new SplitFetcherManager with a single I/O threads.

Parameters:

elementsQueue - The queue that is used to hand over data from the I/O thread (the fetchers) to the reader (which emits the records and book-keeps the state. This must be the same queue instance that is also passed to the SourceReaderBase.

splitReaderSupplier - The factory for the split reader that connects to the source system.

configuration - The configuration to create the fetcher manager.

SingleThreadFetcherManager

@VisibleForTesting
public SingleThreadFetcherManager(FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> elementsQueue,
                                                     java.util.function.Supplier<SplitReader<E,SplitT>> splitReaderSupplier,
                                                     Configuration configuration,
                                                     java.util.function.Consumer<Collection<String>> splitFinishedHook)

Creates a new SplitFetcherManager with a single I/O threads.

Parameters:

elementsQueue - The queue that is used to hand over data from the I/O thread (the fetchers) to the reader (which emits the records and book-keeps the state. This must be the same queue instance that is also passed to the SourceReaderBase.

splitReaderSupplier - The factory for the split reader that connects to the source system.

configuration - The configuration to create the fetcher manager.

splitFinishedHook - Hook for handling finished splits in split fetchers

Method Detail

addSplits

public void addSplits(List<SplitT> splitsToAdd)

Specified by:

addSplits in class SplitFetcherManager<E,SplitT extends SourceSplit>

getRunningFetcher

protected SplitFetcher<E,SplitT> getRunningFetcher()