@Internal public class KafkaShuffleFetcher<T> extends KafkaFetcher<T>
Modifier and Type | Class and Description |
---|---|
static class |
KafkaShuffleFetcher.KafkaShuffleElement
An element in a KafkaShuffle.
|
static class |
KafkaShuffleFetcher.KafkaShuffleElementDeserializer<T>
Deserializer for KafkaShuffleElement.
|
static class |
KafkaShuffleFetcher.KafkaShuffleRecord<T>
One value with Type T in a KafkaShuffle.
|
static class |
KafkaShuffleFetcher.KafkaShuffleWatermark
A watermark element in a KafkaShuffle.
|
checkpointLock, sourceContext, unassignedPartitionsQueue, watermarkOutput
Constructor and Description |
---|
KafkaShuffleFetcher(SourceFunction.SourceContext<T> sourceContext,
Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets,
SerializedValue<WatermarkStrategy<T>> watermarkStrategy,
ProcessingTimeService processingTimeProvider,
long autoWatermarkInterval,
ClassLoader userCodeClassLoader,
String taskNameWithSubtasks,
KafkaDeserializationSchema<T> deserializer,
Properties kafkaProperties,
long pollTimeout,
MetricGroup subtaskMetricGroup,
MetricGroup consumerMetricGroup,
boolean useMetrics,
TypeSerializer<T> typeSerializer,
int producerParallelism) |
Modifier and Type | Method and Description |
---|---|
protected String |
getFetcherName()
Gets the name of this fetcher, for thread naming and logging purposes.
|
protected void |
partitionConsumerRecordsHandler(List<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>> partitionRecords,
KafkaTopicPartitionState<T,org.apache.kafka.common.TopicPartition> partition) |
cancel, createKafkaPartitionHandle, doCommitInternalOffsetsToKafka, runFetchLoop
addDiscoveredPartitions, commitInternalOffsetsToKafka, emitRecordsWithTimestamps, snapshotCurrentState, subscribedPartitionStates
public KafkaShuffleFetcher(SourceFunction.SourceContext<T> sourceContext, Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets, SerializedValue<WatermarkStrategy<T>> watermarkStrategy, ProcessingTimeService processingTimeProvider, long autoWatermarkInterval, ClassLoader userCodeClassLoader, String taskNameWithSubtasks, KafkaDeserializationSchema<T> deserializer, Properties kafkaProperties, long pollTimeout, MetricGroup subtaskMetricGroup, MetricGroup consumerMetricGroup, boolean useMetrics, TypeSerializer<T> typeSerializer, int producerParallelism) throws Exception
Exception
protected String getFetcherName()
KafkaFetcher
getFetcherName
in class KafkaFetcher<T>
protected void partitionConsumerRecordsHandler(List<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>> partitionRecords, KafkaTopicPartitionState<T,org.apache.kafka.common.TopicPartition> partition) throws Exception
partitionConsumerRecordsHandler
in class KafkaFetcher<T>
Exception
Copyright © 2014–2023 The Apache Software Foundation. All rights reserved.