org.apache.flink.streaming.connectors.kafka.internals

Class KafkaTopicPartitionAssigner

java.lang.Object

org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartitionAssigner

@Internal
public class KafkaTopicPartitionAssigner
extends Object

Utility for assigning Kafka partitions to consumer subtasks.

Constructor Summary

Constructors

Constructor and Description
KafkaTopicPartitionAssigner()

Method Summary

Modifier and Type	Method and Description
static int	assign(KafkaTopicPartition partition, int numParallelSubtasks) Returns the index of the target subtask that a specific Kafka partition should be assigned to.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

KafkaTopicPartitionAssigner

public KafkaTopicPartitionAssigner()

Method Detail

assign

public static int assign(KafkaTopicPartition partition,
                         int numParallelSubtasks)

Returns the index of the target subtask that a specific Kafka partition should be assigned to.

The resulting distribution of partitions of a single topic has the following contract:

• 1. Uniformly distributed across subtasks
• 2. Partitions are round-robin distributed (strictly clockwise w.r.t. ascending subtask indices) by using the partition id as the offset from a starting index (i.e., the index of the subtask which partition 0 of the topic will be assigned to, determined using the topic name).

The above contract is crucial and cannot be broken. Consumer subtasks rely on this contract to locally filter out partitions that it should not subscribe to, guaranteeing that all partitions of a single topic will always be assigned to some subtask in a uniformly distributed manner.

Parameters:

partition - the Kafka partition

numParallelSubtasks - total number of parallel subtasks

Returns:

index of the target subtask that the Kafka partition should be assigned to.