@Internal public abstract class AbstractPartitionDiscoverer extends Object
This partition discoverer base class implements the logic around bookkeeping discovered partitions, and using the information to determine whether or not there are new partitions that the consumer subtask should subscribe to.
Subclass implementations should simply implement the logic of using the version-specific Kafka clients to fetch topic and partition metadata.
Since Kafka clients are generally not thread-safe, partition discoverers should not be
concurrently accessed. The only exception for this would be the wakeup()
call, which
allows the discoverer to be interrupted during a discoverPartitions()
call.
Modifier and Type | Class and Description |
---|---|
static class |
AbstractPartitionDiscoverer.ClosedException
Thrown if this discoverer was used to discover partitions after it was closed.
|
static class |
AbstractPartitionDiscoverer.WakeupException
Signaling exception to indicate that an actual Kafka call was interrupted.
|
Constructor and Description |
---|
AbstractPartitionDiscoverer(KafkaTopicsDescriptor topicsDescriptor,
int indexOfThisSubtask,
int numParallelSubtasks) |
Modifier and Type | Method and Description |
---|---|
void |
close()
Closes the partition discoverer, cleaning up all Kafka connections.
|
protected abstract void |
closeConnections()
Close all established connections.
|
List<KafkaTopicPartition> |
discoverPartitions()
Execute a partition discovery attempt for this subtask.
|
protected abstract List<KafkaTopicPartition> |
getAllPartitionsForTopics(List<String> topics)
Fetch the list of all partitions for a specific topics list from Kafka.
|
protected abstract List<String> |
getAllTopics()
Fetch the list of all topics from Kafka.
|
protected abstract void |
initializeConnections()
Establish the required connections in order to fetch topics and partitions metadata.
|
void |
open()
Opens the partition discoverer, initializing all required Kafka connections.
|
boolean |
setAndCheckDiscoveredPartition(KafkaTopicPartition partition)
Sets a partition as discovered.
|
void |
wakeup()
Interrupt an in-progress discovery attempt by throwing a
AbstractPartitionDiscoverer.WakeupException . |
protected abstract void |
wakeupConnections()
Attempt to eagerly wakeup from blocking calls to Kafka in
getAllTopics() and getAllPartitionsForTopics(List) . |
public AbstractPartitionDiscoverer(KafkaTopicsDescriptor topicsDescriptor, int indexOfThisSubtask, int numParallelSubtasks)
public void open() throws Exception
NOTE: thread-safety is not guaranteed.
Exception
public void close() throws Exception
NOTE: thread-safety is not guaranteed.
Exception
public void wakeup()
AbstractPartitionDiscoverer.WakeupException
. If no
attempt is in progress, the immediate next attempt will throw a AbstractPartitionDiscoverer.WakeupException
.
This method can be called concurrently from a different thread.
public List<KafkaTopicPartition> discoverPartitions() throws AbstractPartitionDiscoverer.WakeupException, AbstractPartitionDiscoverer.ClosedException
AbstractPartitionDiscoverer.WakeupException
AbstractPartitionDiscoverer.ClosedException
public boolean setAndCheckDiscoveredPartition(KafkaTopicPartition partition)
If the partition is indeed newly discovered, this method also returns whether the new partition should be subscribed by this subtask.
partition
- the partition to set and checktrue
, if the partition wasn't seen before and should be subscribed by this
subtask; false
otherwiseprotected abstract void initializeConnections() throws Exception
Exception
protected abstract void wakeupConnections()
getAllTopics()
and getAllPartitionsForTopics(List)
.
If the invocation indeed results in interrupting an actual blocking Kafka call, the
implementations of getAllTopics()
and getAllPartitionsForTopics(List)
are responsible of throwing a
AbstractPartitionDiscoverer.WakeupException
.
protected abstract void closeConnections() throws Exception
Exception
protected abstract List<String> getAllTopics() throws AbstractPartitionDiscoverer.WakeupException
protected abstract List<KafkaTopicPartition> getAllPartitionsForTopics(List<String> topics) throws AbstractPartitionDiscoverer.WakeupException
Copyright © 2014–2021 The Apache Software Foundation. All rights reserved.