@Internal public class PulsarOrderedPartitionSplitReader extends Object
PulsarPartitionSplit
, it would be closed once the PulsarOrderedSourceReader
is closed.Modifier and Type | Field and Description |
---|---|
protected SourceReaderMetricGroup |
metricGroup |
protected org.apache.pulsar.client.admin.PulsarAdmin |
pulsarAdmin |
protected org.apache.pulsar.client.api.PulsarClient |
pulsarClient |
protected org.apache.pulsar.client.api.Consumer<byte[]> |
pulsarConsumer |
protected PulsarPartitionSplit |
registeredSplit |
protected SourceConfiguration |
sourceConfiguration |
Constructor and Description |
---|
PulsarOrderedPartitionSplitReader(org.apache.pulsar.client.api.PulsarClient pulsarClient,
org.apache.pulsar.client.admin.PulsarAdmin pulsarAdmin,
SourceConfiguration sourceConfiguration,
SourceReaderMetricGroup metricGroup) |
Modifier and Type | Method and Description |
---|---|
protected void |
afterCreatingConsumer(PulsarPartitionSplit split,
org.apache.pulsar.client.api.Consumer<byte[]> consumer) |
protected void |
beforeCreatingConsumer(PulsarPartitionSplit split) |
void |
close()
Close the split reader.
|
protected org.apache.pulsar.client.api.Consumer<byte[]> |
createPulsarConsumer(PulsarPartitionSplit split)
Create a specified
Consumer by the given split information. |
protected org.apache.pulsar.client.api.Consumer<byte[]> |
createPulsarConsumer(TopicPartition partition)
Create a specified
Consumer by the given topic partition. |
RecordsWithSplitIds<org.apache.pulsar.client.api.Message<byte[]>> |
fetch()
Fetch elements into the blocking queue for the given splits.
|
protected void |
finishedPollMessage(org.apache.pulsar.client.api.Message<byte[]> message) |
void |
handleSplitsChanges(SplitsChange<PulsarPartitionSplit> splitsChanges)
Handle the split changes.
|
void |
notifyCheckpointComplete(TopicPartition partition,
org.apache.pulsar.client.api.MessageId offsetsToCommit) |
void |
pauseOrResumeSplits(Collection<PulsarPartitionSplit> splitsToPause,
Collection<PulsarPartitionSplit> splitsToResume)
Pauses or resumes reading of individual splits readers.
|
protected org.apache.pulsar.client.api.Message<byte[]> |
pollMessage(java.time.Duration timeout) |
void |
wakeUp()
Wake up the split reader in case the fetcher thread is blocking in
SplitReader.fetch() . |
protected final org.apache.pulsar.client.api.PulsarClient pulsarClient
protected final org.apache.pulsar.client.admin.PulsarAdmin pulsarAdmin
protected final SourceConfiguration sourceConfiguration
protected final SourceReaderMetricGroup metricGroup
protected org.apache.pulsar.client.api.Consumer<byte[]> pulsarConsumer
protected PulsarPartitionSplit registeredSplit
public PulsarOrderedPartitionSplitReader(org.apache.pulsar.client.api.PulsarClient pulsarClient, org.apache.pulsar.client.admin.PulsarAdmin pulsarAdmin, SourceConfiguration sourceConfiguration, SourceReaderMetricGroup metricGroup)
protected org.apache.pulsar.client.api.Message<byte[]> pollMessage(java.time.Duration timeout) throws org.apache.pulsar.client.api.PulsarClientException
org.apache.pulsar.client.api.PulsarClientException
protected void finishedPollMessage(org.apache.pulsar.client.api.Message<byte[]> message)
protected void beforeCreatingConsumer(PulsarPartitionSplit split)
public void notifyCheckpointComplete(TopicPartition partition, org.apache.pulsar.client.api.MessageId offsetsToCommit)
public RecordsWithSplitIds<org.apache.pulsar.client.api.Message<byte[]>> fetch() throws IOException
SplitReader
SplitReader.wakeUp()
is invoked. In that case, the
implementation may either decide to return without throwing an exception, or it can just
throw an interrupted exception. In either case, this method should be reentrant, meaning that
the next fetch call should just resume from where the last fetch call was waken up or
interrupted.fetch
in interface SplitReader<org.apache.pulsar.client.api.Message<byte[]>,PulsarPartitionSplit>
IOException
- when encountered IO errors, such as deserialization failures.public void handleSplitsChanges(SplitsChange<PulsarPartitionSplit> splitsChanges)
SplitReader
For the consistency of internal state in SourceReaderBase, if an invalid split is added to
the reader (for example splits without any records), it should be put back into RecordsWithSplitIds
as finished splits so that SourceReaderBase could be able to clean up
resources created for it.
handleSplitsChanges
in interface SplitReader<org.apache.pulsar.client.api.Message<byte[]>,PulsarPartitionSplit>
splitsChanges
- the split changes that the SplitReader needs to handle.public void pauseOrResumeSplits(Collection<PulsarPartitionSplit> splitsToPause, Collection<PulsarPartitionSplit> splitsToResume)
SplitReader
Note that no other methods can be called in parallel, so it's fine to non-atomically update subscriptions. This method is simply providing connectors with more expressive APIs the opportunity to update all subscriptions at once.
This is currently used to align the watermarks of splits, if watermark alignment is used and the source reads from more than one split.
The default implementation throws an UnsupportedOperationException
where the
default implementation will be removed in future releases. To be compatible with future
releases, it is recommended to implement this method and override the default implementation.
pauseOrResumeSplits
in interface SplitReader<org.apache.pulsar.client.api.Message<byte[]>,PulsarPartitionSplit>
splitsToPause
- the splits to pausesplitsToResume
- the splits to resumepublic void wakeUp()
SplitReader
SplitReader.fetch()
.wakeUp
in interface SplitReader<org.apache.pulsar.client.api.Message<byte[]>,PulsarPartitionSplit>
public void close()
SplitReader
close
in interface SplitReader<org.apache.pulsar.client.api.Message<byte[]>,PulsarPartitionSplit>
protected void afterCreatingConsumer(PulsarPartitionSplit split, org.apache.pulsar.client.api.Consumer<byte[]> consumer)
protected org.apache.pulsar.client.api.Consumer<byte[]> createPulsarConsumer(PulsarPartitionSplit split)
Consumer
by the given split information.protected org.apache.pulsar.client.api.Consumer<byte[]> createPulsarConsumer(TopicPartition partition)
Consumer
by the given topic partition.Copyright © 2014–2022 The Apache Software Foundation. All rights reserved.