@Internal public class PulsarUnorderedSourceReader<OUT> extends SourceReaderBase<org.apache.pulsar.client.api.Message<byte[]>,OUT,PulsarPartitionSplit,PulsarPartitionSplitState>
Modifier and Type | Field and Description |
---|---|
protected org.apache.pulsar.client.admin.PulsarAdmin |
pulsarAdmin |
protected org.apache.pulsar.client.api.PulsarClient |
pulsarClient |
protected SourceConfiguration |
sourceConfiguration |
config, context, options, recordEmitter, splitFetcherManager
Constructor and Description |
---|
PulsarUnorderedSourceReader(FutureCompletingBlockingQueue<RecordsWithSplitIds<org.apache.pulsar.client.api.Message<byte[]>>> elementsQueue,
java.util.function.Supplier<PulsarUnorderedPartitionSplitReader> splitReaderSupplier,
PulsarRecordEmitter<OUT> recordEmitter,
SourceReaderContext context,
SourceConfiguration sourceConfiguration,
org.apache.pulsar.client.api.PulsarClient pulsarClient,
org.apache.pulsar.client.admin.PulsarAdmin pulsarAdmin,
org.apache.pulsar.client.api.transaction.TransactionCoordinatorClient coordinatorClient) |
Modifier and Type | Method and Description |
---|---|
void |
addSplits(List<PulsarPartitionSplit> splits)
Adds a list of splits for this reader to read.
|
void |
close() |
protected void |
closeFinishedSplits(Set<String> finishedSplitIds) |
protected PulsarPartitionSplitState |
initializedState(PulsarPartitionSplit split)
When new splits are added to the reader.
|
void |
notifyCheckpointComplete(long checkpointId)
We have an empty default implementation here because most source readers do not have to
implement the method.
|
protected void |
onSplitFinished(Map<String,PulsarPartitionSplitState> finishedSplitIds)
Handles the finished splits to clean the state if needed.
|
void |
pauseOrResumeSplits(Collection<String> splitsToPause,
Collection<String> splitsToResume)
Pauses or resumes reading of individual source splits.
|
List<PulsarPartitionSplit> |
snapshotState(long checkpointId)
Checkpoint on the state of the source.
|
void |
start()
Start the reader.
|
protected PulsarPartitionSplit |
toSplitType(String splitId,
PulsarPartitionSplitState splitState)
Convert a mutable SplitStateT to immutable SplitT.
|
getNumberOfCurrentlyAssignedSplits, handleSourceEvents, isAvailable, notifyNoMoreSplits, pollNext
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
notifyCheckpointAborted
protected final SourceConfiguration sourceConfiguration
protected final org.apache.pulsar.client.api.PulsarClient pulsarClient
protected final org.apache.pulsar.client.admin.PulsarAdmin pulsarAdmin
public PulsarUnorderedSourceReader(FutureCompletingBlockingQueue<RecordsWithSplitIds<org.apache.pulsar.client.api.Message<byte[]>>> elementsQueue, java.util.function.Supplier<PulsarUnorderedPartitionSplitReader> splitReaderSupplier, PulsarRecordEmitter<OUT> recordEmitter, SourceReaderContext context, SourceConfiguration sourceConfiguration, org.apache.pulsar.client.api.PulsarClient pulsarClient, org.apache.pulsar.client.admin.PulsarAdmin pulsarAdmin, @Nullable org.apache.pulsar.client.api.transaction.TransactionCoordinatorClient coordinatorClient)
public void start()
SourceReader
start
in interface SourceReader<OUT,PulsarPartitionSplit>
start
in class SourceReaderBase<org.apache.pulsar.client.api.Message<byte[]>,OUT,PulsarPartitionSplit,PulsarPartitionSplitState>
public void addSplits(List<PulsarPartitionSplit> splits)
SourceReader
SplitEnumeratorContext.assignSplit(SourceSplit, int)
or SplitEnumeratorContext.assignSplits(SplitsAssignment)
.addSplits
in interface SourceReader<OUT,PulsarPartitionSplit>
addSplits
in class SourceReaderBase<org.apache.pulsar.client.api.Message<byte[]>,OUT,PulsarPartitionSplit,PulsarPartitionSplitState>
splits
- The splits assigned by the split enumerator.protected void onSplitFinished(Map<String,PulsarPartitionSplitState> finishedSplitIds)
SourceReaderBase
onSplitFinished
in class SourceReaderBase<org.apache.pulsar.client.api.Message<byte[]>,OUT,PulsarPartitionSplit,PulsarPartitionSplitState>
public List<PulsarPartitionSplit> snapshotState(long checkpointId)
SourceReader
snapshotState
in interface SourceReader<OUT,PulsarPartitionSplit>
snapshotState
in class SourceReaderBase<org.apache.pulsar.client.api.Message<byte[]>,OUT,PulsarPartitionSplit,PulsarPartitionSplitState>
public void notifyCheckpointComplete(long checkpointId) throws Exception
SourceReader
checkpointId
- The ID of the checkpoint that has been completed.Exception
- This method can propagate exceptions, which leads to a failure/recovery for
the task. Note that this will NOT lead to the checkpoint being revoked.CheckpointListener.notifyCheckpointComplete(long)
public void close() throws Exception
close
in interface AutoCloseable
Exception
protected PulsarPartitionSplitState initializedState(PulsarPartitionSplit split)
SourceReaderBase
initializedState
in class SourceReaderBase<org.apache.pulsar.client.api.Message<byte[]>,OUT,PulsarPartitionSplit,PulsarPartitionSplitState>
split
- a newly added split.protected PulsarPartitionSplit toSplitType(String splitId, PulsarPartitionSplitState splitState)
SourceReaderBase
toSplitType
in class SourceReaderBase<org.apache.pulsar.client.api.Message<byte[]>,OUT,PulsarPartitionSplit,PulsarPartitionSplitState>
splitState
- splitState.public void pauseOrResumeSplits(Collection<String> splitsToPause, Collection<String> splitsToResume)
SourceReader
Note that no other methods can be called in parallel, so updating subscriptions can be done atomically. This method is simply providing connectors with more expressive APIs the opportunity to update all subscriptions at once.
This is currently used to align the watermarks of splits, if watermark alignment is used and the source reads from more than one split.
The default implementation throws an UnsupportedOperationException
where the
default implementation will be removed in future releases. To be compatible with future
releases, it is recommended to implement this method and override the default implementation.
pauseOrResumeSplits
in interface SourceReader<OUT,PulsarPartitionSplit>
pauseOrResumeSplits
in class SourceReaderBase<org.apache.pulsar.client.api.Message<byte[]>,OUT,PulsarPartitionSplit,PulsarPartitionSplitState>
splitsToPause
- the splits to pausesplitsToResume
- the splits to resumeCopyright © 2014–2022 The Apache Software Foundation. All rights reserved.