@Internal public class KafkaSourceEnumerator extends Object implements SplitEnumerator<KafkaPartitionSplit,KafkaSourceEnumState>
Modifier and Type | Class and Description |
---|---|
static class |
KafkaSourceEnumerator.PartitionOffsetsRetrieverImpl
The implementation for offsets retriever with a consumer and an admin client.
|
Constructor and Description |
---|
KafkaSourceEnumerator(KafkaSubscriber subscriber,
OffsetsInitializer startingOffsetInitializer,
OffsetsInitializer stoppingOffsetInitializer,
Properties properties,
SplitEnumeratorContext<KafkaPartitionSplit> context,
Boundedness boundedness) |
KafkaSourceEnumerator(KafkaSubscriber subscriber,
OffsetsInitializer startingOffsetInitializer,
OffsetsInitializer stoppingOffsetInitializer,
Properties properties,
SplitEnumeratorContext<KafkaPartitionSplit> context,
Boundedness boundedness,
Set<org.apache.kafka.common.TopicPartition> assignedPartitions) |
Modifier and Type | Method and Description |
---|---|
void |
addReader(int subtaskId)
Add a new source reader with the given subtask ID.
|
void |
addSplitsBack(List<KafkaPartitionSplit> splits,
int subtaskId)
Add splits back to the split enumerator.
|
void |
close()
Called to close the enumerator, in case it holds on to any resources, like threads or network
connections.
|
void |
handleSplitRequest(int subtaskId,
String requesterHostname)
Handles the request for a split.
|
KafkaSourceEnumState |
snapshotState(long checkpointId)
Creates a snapshot of the state of this split enumerator, to be stored in a checkpoint.
|
void |
start()
Start the enumerator.
|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
handleSourceEvent, notifyCheckpointComplete
notifyCheckpointAborted
public KafkaSourceEnumerator(KafkaSubscriber subscriber, OffsetsInitializer startingOffsetInitializer, OffsetsInitializer stoppingOffsetInitializer, Properties properties, SplitEnumeratorContext<KafkaPartitionSplit> context, Boundedness boundedness)
public KafkaSourceEnumerator(KafkaSubscriber subscriber, OffsetsInitializer startingOffsetInitializer, OffsetsInitializer stoppingOffsetInitializer, Properties properties, SplitEnumeratorContext<KafkaPartitionSplit> context, Boundedness boundedness, Set<org.apache.kafka.common.TopicPartition> assignedPartitions)
public void start()
Depending on partitionDiscoveryIntervalMs
, the enumerator will trigger a one-time
partition discovery, or schedule a callable for discover partitions periodically.
The invoking chain of partition discovery would be:
getSubscribedTopicPartitions()
in worker thread
checkPartitionChanges(java.util.Set<org.apache.kafka.common.TopicPartition>, java.lang.Throwable)
in coordinator thread
initializePartitionSplits(org.apache.flink.connector.kafka.source.enumerator.KafkaSourceEnumerator.PartitionChange)
in worker thread
handlePartitionSplitChanges(org.apache.flink.connector.kafka.source.enumerator.KafkaSourceEnumerator.PartitionSplitChange, java.lang.Throwable)
in coordinator thread
start
in interface SplitEnumerator<KafkaPartitionSplit,KafkaSourceEnumState>
public void handleSplitRequest(int subtaskId, @Nullable String requesterHostname)
SplitEnumerator
SourceReaderContext.sendSplitRequest()
method.handleSplitRequest
in interface SplitEnumerator<KafkaPartitionSplit,KafkaSourceEnumState>
subtaskId
- the subtask id of the source reader who sent the source event.requesterHostname
- Optional, the hostname where the requesting task is running. This
can be used to make split assignments locality-aware.public void addSplitsBack(List<KafkaPartitionSplit> splits, int subtaskId)
SplitEnumerator
SourceReader
fails and there are splits assigned to it after the last successful checkpoint.addSplitsBack
in interface SplitEnumerator<KafkaPartitionSplit,KafkaSourceEnumState>
splits
- The splits to add back to the enumerator for reassignment.subtaskId
- The id of the subtask to which the returned splits belong.public void addReader(int subtaskId)
SplitEnumerator
addReader
in interface SplitEnumerator<KafkaPartitionSplit,KafkaSourceEnumState>
subtaskId
- the subtask ID of the new source reader.public KafkaSourceEnumState snapshotState(long checkpointId) throws Exception
SplitEnumerator
The snapshot should contain the latest state of the enumerator: It should assume that all
operations that happened before the snapshot have successfully completed. For example all
splits assigned to readers via SplitEnumeratorContext.assignSplit(SourceSplit, int)
and SplitEnumeratorContext.assignSplits(SplitsAssignment)
) don't need to be included
in the snapshot anymore.
This method takes the ID of the checkpoint for which the state is snapshotted. Most implementations should be able to ignore this parameter, because for the contents of the snapshot, it doesn't matter for which checkpoint it gets created. This parameter can be interesting for source connectors with external systems where those systems are themselves aware of checkpoints; for example in cases where the enumerator notifies that system about a specific checkpoint being triggered.
snapshotState
in interface SplitEnumerator<KafkaPartitionSplit,KafkaSourceEnumState>
checkpointId
- The ID of the checkpoint for which the snapshot is created.Exception
- when the snapshot cannot be taken.public void close()
SplitEnumerator
close
in interface AutoCloseable
close
in interface SplitEnumerator<KafkaPartitionSplit,KafkaSourceEnumState>
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.