OUT
- the type of the pulsar source message that would be serialized to downstream.@Internal public class PulsarOrderedPartitionSplitReader<OUT> extends Object
PulsarPartitionSplit
, it would be closed once the PulsarOrderedSourceReader
is closed.Modifier and Type | Field and Description |
---|---|
protected Configuration |
configuration |
protected PulsarDeserializationSchema<OUT> |
deserializationSchema |
protected org.apache.pulsar.client.admin.PulsarAdmin |
pulsarAdmin |
protected org.apache.pulsar.client.api.PulsarClient |
pulsarClient |
protected org.apache.pulsar.client.api.Consumer<byte[]> |
pulsarConsumer |
protected PulsarPartitionSplit |
registeredSplit |
protected SourceConfiguration |
sourceConfiguration |
Constructor and Description |
---|
PulsarOrderedPartitionSplitReader(org.apache.pulsar.client.api.PulsarClient pulsarClient,
org.apache.pulsar.client.admin.PulsarAdmin pulsarAdmin,
Configuration configuration,
SourceConfiguration sourceConfiguration,
PulsarDeserializationSchema<OUT> deserializationSchema) |
Modifier and Type | Method and Description |
---|---|
protected void |
afterCreatingConsumer(PulsarPartitionSplit split,
org.apache.pulsar.client.api.Consumer<byte[]> consumer) |
protected void |
beforeCreatingConsumer(PulsarPartitionSplit split) |
void |
close()
Close the split reader.
|
protected org.apache.pulsar.client.api.Consumer<byte[]> |
createPulsarConsumer(PulsarPartitionSplit split)
Create a specified
Consumer by the given split information. |
protected org.apache.pulsar.client.api.Consumer<byte[]> |
createPulsarConsumer(TopicPartition partition)
Create a specified
Consumer by the given topic partition. |
RecordsWithSplitIds<PulsarMessage<OUT>> |
fetch()
Fetch elements into the blocking queue for the given splits.
|
protected void |
finishedPollMessage(org.apache.pulsar.client.api.Message<byte[]> message) |
void |
handleSplitsChanges(SplitsChange<PulsarPartitionSplit> splitsChanges)
Handle the split changes.
|
void |
notifyCheckpointComplete(TopicPartition partition,
org.apache.pulsar.client.api.MessageId offsetsToCommit) |
protected org.apache.pulsar.client.api.Message<byte[]> |
pollMessage(java.time.Duration timeout) |
void |
wakeUp()
Wake up the split reader in case the fetcher thread is blocking in
SplitReader.fetch() . |
protected final org.apache.pulsar.client.api.PulsarClient pulsarClient
protected final org.apache.pulsar.client.admin.PulsarAdmin pulsarAdmin
protected final Configuration configuration
protected final SourceConfiguration sourceConfiguration
protected final PulsarDeserializationSchema<OUT> deserializationSchema
protected org.apache.pulsar.client.api.Consumer<byte[]> pulsarConsumer
protected PulsarPartitionSplit registeredSplit
public PulsarOrderedPartitionSplitReader(org.apache.pulsar.client.api.PulsarClient pulsarClient, org.apache.pulsar.client.admin.PulsarAdmin pulsarAdmin, Configuration configuration, SourceConfiguration sourceConfiguration, PulsarDeserializationSchema<OUT> deserializationSchema)
protected org.apache.pulsar.client.api.Message<byte[]> pollMessage(java.time.Duration timeout) throws org.apache.pulsar.client.api.PulsarClientException
org.apache.pulsar.client.api.PulsarClientException
protected void finishedPollMessage(org.apache.pulsar.client.api.Message<byte[]> message)
protected void beforeCreatingConsumer(PulsarPartitionSplit split)
public void notifyCheckpointComplete(TopicPartition partition, org.apache.pulsar.client.api.MessageId offsetsToCommit)
public RecordsWithSplitIds<PulsarMessage<OUT>> fetch() throws IOException
SplitReader
SplitReader.wakeUp()
is invoked. In that case, the
implementation may either decide to return without throwing an exception, or it can just
throw an interrupted exception. In either case, this method should be reentrant, meaning that
the next fetch call should just resume from where the last fetch call was waken up or
interrupted.fetch
in interface SplitReader<PulsarMessage<OUT>,PulsarPartitionSplit>
IOException
- when encountered IO errors, such as deserialization failures.public void handleSplitsChanges(SplitsChange<PulsarPartitionSplit> splitsChanges)
SplitReader
For the consistency of internal state in SourceReaderBase, if an invalid split is added to
the reader (for example splits without any records), it should be put back into RecordsWithSplitIds
as finished splits so that SourceReaderBase could be able to clean up
resources created for it.
handleSplitsChanges
in interface SplitReader<PulsarMessage<OUT>,PulsarPartitionSplit>
splitsChanges
- the split changes that the SplitReader needs to handle.public void wakeUp()
SplitReader
SplitReader.fetch()
.wakeUp
in interface SplitReader<PulsarMessage<OUT>,PulsarPartitionSplit>
public void close()
SplitReader
close
in interface SplitReader<PulsarMessage<OUT>,PulsarPartitionSplit>
protected void afterCreatingConsumer(PulsarPartitionSplit split, org.apache.pulsar.client.api.Consumer<byte[]> consumer)
protected org.apache.pulsar.client.api.Consumer<byte[]> createPulsarConsumer(PulsarPartitionSplit split)
Consumer
by the given split information.protected org.apache.pulsar.client.api.Consumer<byte[]> createPulsarConsumer(TopicPartition partition)
Consumer
by the given topic partition.Copyright © 2014–2023 The Apache Software Foundation. All rights reserved.