@Internal public abstract class PulsarFetcherManagerBase extends SplitFetcherManager<org.apache.pulsar.client.api.Message<byte[]>,PulsarPartitionSplit>
fetchers
Modifier | Constructor and Description |
---|---|
protected |
PulsarFetcherManagerBase(FutureCompletingBlockingQueue<RecordsWithSplitIds<org.apache.pulsar.client.api.Message<byte[]>>> elementsQueue,
java.util.function.Supplier<SplitReader<org.apache.pulsar.client.api.Message<byte[]>,PulsarPartitionSplit>> splitReaderSupplier,
Configuration configuration)
Creates a new SplitFetcherManager with multiple I/O threads.
|
Modifier and Type | Method and Description |
---|---|
void |
addSplits(List<PulsarPartitionSplit> splitsToAdd)
Override this method for supporting multiple thread fetching, one fetcher thread for one
split.
|
void |
closeFetcher(String splitId)
Close the finished split related fetcher.
|
protected SplitFetcher<org.apache.pulsar.client.api.Message<byte[]>,PulsarPartitionSplit> |
getOrCreateFetcher(String splitId) |
protected void |
startFetcher(SplitFetcher<org.apache.pulsar.client.api.Message<byte[]>,PulsarPartitionSplit> fetcher) |
checkErrors, close, createSplitFetcher, getNumAliveFetchers, maybeShutdownFinishedFetchers, pauseOrResumeSplits
protected PulsarFetcherManagerBase(FutureCompletingBlockingQueue<RecordsWithSplitIds<org.apache.pulsar.client.api.Message<byte[]>>> elementsQueue, java.util.function.Supplier<SplitReader<org.apache.pulsar.client.api.Message<byte[]>,PulsarPartitionSplit>> splitReaderSupplier, Configuration configuration)
elementsQueue
- The queue that is used to hand over data from the I/O thread (the
fetchers) to the reader (which emits the records and book-keeps the state. This must be
the same queue instance that is also passed to the SourceReaderBase
.splitReaderSupplier
- The factory for the split reader that connects to the sourcepublic void addSplits(List<PulsarPartitionSplit> splitsToAdd)
addSplits
in class SplitFetcherManager<org.apache.pulsar.client.api.Message<byte[]>,PulsarPartitionSplit>
protected void startFetcher(SplitFetcher<org.apache.pulsar.client.api.Message<byte[]>,PulsarPartitionSplit> fetcher)
startFetcher
in class SplitFetcherManager<org.apache.pulsar.client.api.Message<byte[]>,PulsarPartitionSplit>
public void closeFetcher(String splitId)
protected SplitFetcher<org.apache.pulsar.client.api.Message<byte[]>,PulsarPartitionSplit> getOrCreateFetcher(String splitId)
Copyright © 2014–2022 The Apache Software Foundation. All rights reserved.