org.apache.flink.connector.kafka.source.reader

Class KafkaPartitionSplitReader

java.lang.Object

org.apache.flink.connector.kafka.source.reader.KafkaPartitionSplitReader

All Implemented Interfaces:

SplitReader<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,KafkaPartitionSplit>

@Internal
public class KafkaPartitionSplitReader
extends Object
implements SplitReader<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,KafkaPartitionSplit>

A SplitReader implementation that reads records from Kafka partitions.

Constructor Summary

Constructors

Constructor and Description
KafkaPartitionSplitReader(Properties props, SourceReaderContext context, KafkaSourceReaderMetrics kafkaSourceReaderMetrics)

Method Summary

Modifier and Type	Method and Description
void	close() Close the split reader.
RecordsWithSplitIds<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>>	fetch() Fetch elements into the blocking queue for the given splits.
void	handleSplitsChanges(SplitsChange<KafkaPartitionSplit> splitsChange) Handle the split changes.
void	notifyCheckpointComplete(Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> offsetsToCommit, org.apache.kafka.clients.consumer.OffsetCommitCallback offsetCommitCallback)
void	wakeUp() Wake up the split reader in case the fetcher thread is blocking in SplitReader.fetch().

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

KafkaPartitionSplitReader

public KafkaPartitionSplitReader(Properties props,
                                 SourceReaderContext context,
                                 KafkaSourceReaderMetrics kafkaSourceReaderMetrics)

Method Detail

fetch

public RecordsWithSplitIds<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>> fetch()
                                                                                           throws IOException

Description copied from interface: SplitReader

Fetch elements into the blocking queue for the given splits. The fetch call could be blocking but it should get unblocked when SplitReader.wakeUp() is invoked. In that case, the implementation may either decide to return without throwing an exception, or it can just throw an interrupted exception. In either case, this method should be reentrant, meaning that the next fetch call should just resume from where the last fetch call was waken up or interrupted.

Specified by:

fetch in interface SplitReader<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,KafkaPartitionSplit>

Returns:

the Ids of the finished splits.

Throws:

IOException - when encountered IO errors, such as deserialization failures.

handleSplitsChanges

public void handleSplitsChanges(SplitsChange<KafkaPartitionSplit> splitsChange)

Description copied from interface: SplitReader

Handle the split changes. This call should be non-blocking.

For the consistency of internal state in SourceReaderBase, if an invalid split is added to the reader (for example splits without any records), it should be put back into RecordsWithSplitIds as finished splits so that SourceReaderBase could be able to clean up resources created for it.

Specified by:

handleSplitsChanges in interface SplitReader<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,KafkaPartitionSplit>

Parameters:

splitsChange - the split changes that the SplitReader needs to handle.

wakeUp

public void wakeUp()

Description copied from interface: SplitReader

Wake up the split reader in case the fetcher thread is blocking in SplitReader.fetch().

Specified by:

wakeUp in interface SplitReader<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,KafkaPartitionSplit>

close

public void close()
           throws Exception

Description copied from interface: SplitReader

Close the split reader.

Specified by:

close in interface SplitReader<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,KafkaPartitionSplit>

Throws:

Exception - if closing the split reader failed.

notifyCheckpointComplete

public void notifyCheckpointComplete(Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> offsetsToCommit,
                                     org.apache.kafka.clients.consumer.OffsetCommitCallback offsetCommitCallback)