Modifier and Type | Field and Description |
---|---|
protected Map<KafkaTopicPartition,Long> |
FlinkKafkaConsumerBase.specificStartupOffsets
Specific startup offsets; only relevant when startup mode is
StartupMode.SPECIFIC_OFFSETS |
Modifier and Type | Method and Description |
---|---|
protected abstract List<KafkaTopicPartition> |
FlinkKafkaConsumerBase.getKafkaPartitions(List<String> topics) |
protected List<KafkaTopicPartition> |
FlinkKafkaConsumer09.getKafkaPartitions(List<String> topics) |
protected List<KafkaTopicPartition> |
FlinkKafkaConsumer08.getKafkaPartitions(List<String> topics) |
protected static Map<KafkaTopicPartition,Long> |
FlinkKafkaConsumerBase.initializeSubscribedPartitionsToStartOffsets(List<KafkaTopicPartition> kafkaTopicPartitions,
int indexOfThisSubtask,
int numParallelSubtasks,
StartupMode startupMode,
Map<KafkaTopicPartition,Long> specificStartupOffsets)
Initializes
FlinkKafkaConsumerBase.subscribedPartitionsToStartOffsets with appropriate
values. |
Modifier and Type | Method and Description |
---|---|
protected AbstractFetcher<T,?> |
FlinkKafkaConsumer010.createFetcher(SourceFunction.SourceContext<T> sourceContext,
Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets,
SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,
SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,
StreamingRuntimeContext runtimeContext,
OffsetCommitMode offsetCommitMode) |
protected abstract AbstractFetcher<T,?> |
FlinkKafkaConsumerBase.createFetcher(SourceFunction.SourceContext<T> sourceContext,
Map<KafkaTopicPartition,Long> subscribedPartitionsToStartOffsets,
SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,
SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,
StreamingRuntimeContext runtimeContext,
OffsetCommitMode offsetCommitMode)
Creates the fetcher that connect to the Kafka brokers, pulls data, deserialized the
data, and emits it into the data streams.
|
protected AbstractFetcher<T,?> |
FlinkKafkaConsumer09.createFetcher(SourceFunction.SourceContext<T> sourceContext,
Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets,
SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,
SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,
StreamingRuntimeContext runtimeContext,
OffsetCommitMode offsetCommitMode) |
protected AbstractFetcher<T,?> |
FlinkKafkaConsumer08.createFetcher(SourceFunction.SourceContext<T> sourceContext,
Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets,
SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,
SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,
StreamingRuntimeContext runtimeContext,
OffsetCommitMode offsetCommitMode) |
protected static Map<KafkaTopicPartition,Long> |
FlinkKafkaConsumerBase.initializeSubscribedPartitionsToStartOffsets(List<KafkaTopicPartition> kafkaTopicPartitions,
int indexOfThisSubtask,
int numParallelSubtasks,
StartupMode startupMode,
Map<KafkaTopicPartition,Long> specificStartupOffsets)
Initializes
FlinkKafkaConsumerBase.subscribedPartitionsToStartOffsets with appropriate
values. |
protected static Map<KafkaTopicPartition,Long> |
FlinkKafkaConsumerBase.initializeSubscribedPartitionsToStartOffsets(List<KafkaTopicPartition> kafkaTopicPartitions,
int indexOfThisSubtask,
int numParallelSubtasks,
StartupMode startupMode,
Map<KafkaTopicPartition,Long> specificStartupOffsets)
Initializes
FlinkKafkaConsumerBase.subscribedPartitionsToStartOffsets with appropriate
values. |
protected static void |
FlinkKafkaConsumerBase.logPartitionInfo(org.slf4j.Logger logger,
List<KafkaTopicPartition> partitionInfos)
Logs the partition information in INFO level.
|
void |
FlinkKafkaConsumerBase.restoreState(HashMap<KafkaTopicPartition,Long> restoredOffsets) |
FlinkKafkaConsumerBase<T> |
FlinkKafkaConsumerBase.setStartFromSpecificOffsets(Map<KafkaTopicPartition,Long> specificStartupOffsets)
Specifies the consumer to start reading partitions from specific offsets, set independently for each partition.
|
Modifier and Type | Method and Description |
---|---|
org.apache.kafka.common.TopicPartition |
Kafka09Fetcher.createKafkaPartitionHandle(KafkaTopicPartition partition) |
Modifier and Type | Method and Description |
---|---|
void |
Kafka09Fetcher.commitInternalOffsetsToKafka(Map<KafkaTopicPartition,Long> offsets,
KafkaCommitCallback commitCallback) |
Constructor and Description |
---|
Kafka010Fetcher(SourceFunction.SourceContext<T> sourceContext,
Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets,
SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,
SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,
ProcessingTimeService processingTimeProvider,
long autoWatermarkInterval,
ClassLoader userCodeClassLoader,
String taskNameWithSubtasks,
MetricGroup metricGroup,
KeyedDeserializationSchema<T> deserializer,
Properties kafkaProperties,
long pollTimeout,
boolean useMetrics) |
Kafka09Fetcher(SourceFunction.SourceContext<T> sourceContext,
Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets,
SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,
SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,
ProcessingTimeService processingTimeProvider,
long autoWatermarkInterval,
ClassLoader userCodeClassLoader,
String taskNameWithSubtasks,
MetricGroup metricGroup,
KeyedDeserializationSchema<T> deserializer,
Properties kafkaProperties,
long pollTimeout,
boolean useMetrics) |
Modifier and Type | Method and Description |
---|---|
KafkaTopicPartition |
KafkaTopicPartitionState.getKafkaTopicPartition()
Gets Flink's descriptor for the Kafka Partition.
|
KafkaTopicPartition |
KafkaTopicPartitionLeader.getTopicPartition() |
Modifier and Type | Method and Description |
---|---|
static List<KafkaTopicPartition> |
KafkaTopicPartition.dropLeaderData(List<KafkaTopicPartitionLeader> partitionInfos) |
HashMap<KafkaTopicPartition,Long> |
AbstractFetcher.snapshotCurrentState()
Takes a snapshot of the partition offsets.
|
Modifier and Type | Method and Description |
---|---|
static int |
KafkaTopicPartitionAssigner.assign(KafkaTopicPartition partition,
int numParallelSubtasks)
Returns the index of the target subtask that a specific Kafka partition should be
assigned to.
|
abstract KPH |
AbstractFetcher.createKafkaPartitionHandle(KafkaTopicPartition partition)
Creates the Kafka version specific representation of the given
topic partition.
|
kafka.common.TopicAndPartition |
Kafka08Fetcher.createKafkaPartitionHandle(KafkaTopicPartition partition) |
Long |
ZookeeperOffsetHandler.getCommittedOffset(KafkaTopicPartition partition) |
Modifier and Type | Method and Description |
---|---|
abstract void |
AbstractFetcher.commitInternalOffsetsToKafka(Map<KafkaTopicPartition,Long> offsets,
KafkaCommitCallback commitCallback)
Commits the given partition offsets to the Kafka brokers (or to ZooKeeper for
older Kafka versions).
|
void |
Kafka08Fetcher.commitInternalOffsetsToKafka(Map<KafkaTopicPartition,Long> offsets,
KafkaCommitCallback commitCallback) |
void |
ZookeeperOffsetHandler.prepareAndCommitOffsets(Map<KafkaTopicPartition,Long> internalOffsets)
Commits offsets for Kafka partitions to ZooKeeper.
|
static String |
KafkaTopicPartition.toString(List<KafkaTopicPartition> partitions) |
static String |
KafkaTopicPartition.toString(Map<KafkaTopicPartition,Long> map) |
Constructor and Description |
---|
KafkaTopicPartitionLeader(KafkaTopicPartition topicPartition,
org.apache.kafka.common.Node leader) |
KafkaTopicPartitionState(KafkaTopicPartition partition,
KPH kafkaPartitionHandle) |
KafkaTopicPartitionStateWithPeriodicWatermarks(KafkaTopicPartition partition,
KPH kafkaPartitionHandle,
AssignerWithPeriodicWatermarks<T> timestampsAndWatermarks) |
KafkaTopicPartitionStateWithPunctuatedWatermarks(KafkaTopicPartition partition,
KPH kafkaPartitionHandle,
AssignerWithPunctuatedWatermarks<T> timestampsAndWatermarks) |
Constructor and Description |
---|
AbstractFetcher(SourceFunction.SourceContext<T> sourceContext,
Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets,
SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,
SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,
ProcessingTimeService processingTimeProvider,
long autoWatermarkInterval,
ClassLoader userCodeClassLoader,
boolean useMetrics) |
Kafka08Fetcher(SourceFunction.SourceContext<T> sourceContext,
Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets,
SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,
SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,
StreamingRuntimeContext runtimeContext,
KeyedDeserializationSchema<T> deserializer,
Properties kafkaProperties,
long autoCommitInterval,
boolean useMetrics) |
Copyright © 2014–2018 The Apache Software Foundation. All rights reserved.