Modifier and Type | Field and Description |
---|---|
protected List<KafkaTopicPartition> |
FlinkKafkaConsumerBase.allSubscribedPartitions
The set of topic partitions that the source will read
|
Modifier and Type | Method and Description |
---|---|
protected static List<KafkaTopicPartition> |
FlinkKafkaConsumerBase.assignPartitions(Map<KafkaTopicPartition,Long> restoredPartitionOffsets,
List<KafkaTopicPartition> completeKafkaPartitionsList,
int numConsumers,
int consumerIndex)
Determines which partitions the consumer should subscribe to.
|
HashMap<KafkaTopicPartition,Long> |
FlinkKafkaConsumerBase.snapshotState(long checkpointId,
long checkpointTimestamp) |
Modifier and Type | Method and Description |
---|---|
protected static List<KafkaTopicPartition> |
FlinkKafkaConsumerBase.assignPartitions(Map<KafkaTopicPartition,Long> restoredPartitionOffsets,
List<KafkaTopicPartition> completeKafkaPartitionsList,
int numConsumers,
int consumerIndex)
Determines which partitions the consumer should subscribe to.
|
protected static List<KafkaTopicPartition> |
FlinkKafkaConsumerBase.assignPartitions(Map<KafkaTopicPartition,Long> restoredPartitionOffsets,
List<KafkaTopicPartition> completeKafkaPartitionsList,
int numConsumers,
int consumerIndex)
Determines which partitions the consumer should subscribe to.
|
protected AbstractFetcher<T,?> |
FlinkKafkaConsumer09.createFetcher(SourceFunction.SourceContext<T> sourceContext,
List<KafkaTopicPartition> thisSubtaskPartitions,
SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,
SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,
StreamingRuntimeContext runtimeContext) |
protected abstract AbstractFetcher<T,?> |
FlinkKafkaConsumerBase.createFetcher(SourceFunction.SourceContext<T> sourceContext,
List<KafkaTopicPartition> thisSubtaskPartitions,
SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,
SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,
StreamingRuntimeContext runtimeContext)
Creates the fetcher that connect to the Kafka brokers, pulls data, deserialized the
data, and emits it into the data streams.
|
protected AbstractFetcher<T,?> |
FlinkKafkaConsumer08.createFetcher(SourceFunction.SourceContext<T> sourceContext,
List<KafkaTopicPartition> thisSubtaskPartitions,
SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,
SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,
StreamingRuntimeContext runtimeContext) |
protected static void |
FlinkKafkaConsumerBase.logPartitionInfo(org.slf4j.Logger logger,
List<KafkaTopicPartition> partitionInfos)
Logs the partition information in INFO level.
|
void |
FlinkKafkaConsumerBase.restoreState(HashMap<KafkaTopicPartition,Long> restoredOffsets) |
protected void |
FlinkKafkaConsumerBase.setSubscribedPartitions(List<KafkaTopicPartition> allSubscribedPartitions)
This method must be called from the subclasses, to set the list of all subscribed partitions
that this consumer will fetch from (across all subtasks).
|
Modifier and Type | Method and Description |
---|---|
org.apache.kafka.common.TopicPartition |
Kafka09Fetcher.createKafkaPartitionHandle(KafkaTopicPartition partition) |
Modifier and Type | Method and Description |
---|---|
void |
Kafka09Fetcher.commitSpecificOffsetsToKafka(Map<KafkaTopicPartition,Long> offsets) |
Constructor and Description |
---|
Kafka09Fetcher(SourceFunction.SourceContext<T> sourceContext,
List<KafkaTopicPartition> assignedPartitions,
SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,
SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,
StreamingRuntimeContext runtimeContext,
KeyedDeserializationSchema<T> deserializer,
Properties kafkaProperties,
long pollTimeout,
boolean useMetrics) |
Modifier and Type | Method and Description |
---|---|
KafkaTopicPartition |
KafkaTopicPartitionState.getKafkaTopicPartition()
Gets Flink's descriptor for the Kafka Partition.
|
KafkaTopicPartition |
KafkaTopicPartitionLeader.getTopicPartition() |
Modifier and Type | Method and Description |
---|---|
static List<KafkaTopicPartition> |
KafkaTopicPartition.dropLeaderData(List<KafkaTopicPartitionLeader> partitionInfos) |
Map<KafkaTopicPartition,Long> |
ZookeeperOffsetHandler.getOffsets(List<KafkaTopicPartition> partitions) |
HashMap<KafkaTopicPartition,Long> |
AbstractFetcher.snapshotCurrentState()
Takes a snapshot of the partition offsets.
|
Modifier and Type | Method and Description |
---|---|
abstract KPH |
AbstractFetcher.createKafkaPartitionHandle(KafkaTopicPartition partition)
Creates the Kafka version specific representation of the given
topic partition.
|
kafka.common.TopicAndPartition |
Kafka08Fetcher.createKafkaPartitionHandle(KafkaTopicPartition partition) |
Modifier and Type | Method and Description |
---|---|
abstract void |
AbstractFetcher.commitSpecificOffsetsToKafka(Map<KafkaTopicPartition,Long> offsets)
Commits the given partition offsets to the Kafka brokers (or to ZooKeeper for
older Kafka versions).
|
void |
Kafka08Fetcher.commitSpecificOffsetsToKafka(Map<KafkaTopicPartition,Long> offsets) |
Map<KafkaTopicPartition,Long> |
ZookeeperOffsetHandler.getOffsets(List<KafkaTopicPartition> partitions) |
void |
AbstractFetcher.restoreOffsets(Map<KafkaTopicPartition,Long> restoredOffsets)
Restores the partition offsets.
|
static String |
KafkaTopicPartition.toString(List<KafkaTopicPartition> partitions) |
static String |
KafkaTopicPartition.toString(Map<KafkaTopicPartition,Long> map) |
void |
ZookeeperOffsetHandler.writeOffsets(Map<KafkaTopicPartition,Long> offsetsToWrite)
Writes given set of offsets for Kafka partitions to ZooKeeper.
|
Constructor and Description |
---|
KafkaTopicPartitionLeader(KafkaTopicPartition topicPartition,
org.apache.kafka.common.Node leader) |
KafkaTopicPartitionState(KafkaTopicPartition partition,
KPH kafkaPartitionHandle) |
KafkaTopicPartitionStateWithPeriodicWatermarks(KafkaTopicPartition partition,
KPH kafkaPartitionHandle,
AssignerWithPeriodicWatermarks<T> timestampsAndWatermarks) |
KafkaTopicPartitionStateWithPunctuatedWatermarks(KafkaTopicPartition partition,
KPH kafkaPartitionHandle,
AssignerWithPunctuatedWatermarks<T> timestampsAndWatermarks) |
Constructor and Description |
---|
AbstractFetcher(SourceFunction.SourceContext<T> sourceContext,
List<KafkaTopicPartition> assignedPartitions,
SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,
SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,
StreamingRuntimeContext runtimeContext,
boolean useMetrics) |
Kafka08Fetcher(SourceFunction.SourceContext<T> sourceContext,
List<KafkaTopicPartition> assignedPartitions,
SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,
SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,
StreamingRuntimeContext runtimeContext,
KeyedDeserializationSchema<T> deserializer,
Properties kafkaProperties,
long invalidOffsetBehavior,
long autoCommitInterval,
boolean useMetrics) |
Copyright © 2014–2017 The Apache Software Foundation. All rights reserved.