org.apache.flink.connector.kafka.source

Class KafkaSource<OUT>

java.lang.Object

org.apache.flink.connector.kafka.source.KafkaSource<OUT>

Type Parameters:

OUT - the output type of the source.

All Implemented Interfaces:

Serializable, Source<OUT,KafkaPartitionSplit,KafkaSourceEnumState>, ResultTypeQueryable<OUT>

@PublicEvolving
public class KafkaSource<OUT>
extends Object
implements Source<OUT,KafkaPartitionSplit,KafkaSourceEnumState>, ResultTypeQueryable<OUT>

The Source implementation of Kafka. Please use a KafkaSourceBuilder to construct a KafkaSource. The following example shows how to create a KafkaSource emitting records of String type.

 KafkaSource<String> source = KafkaSource
     .<String>builder()
     .setBootstrapServers(KafkaSourceTestEnv.brokerConnectionStrings)
     .setGroupId("MyGroup")
     .setTopics(Arrays.asList(TOPIC1, TOPIC2))
     .setDeserializer(new TestingKafkaRecordDeserializationSchema())
     .setStartingOffsets(OffsetsInitializer.earliest())
     .build();
 

KafkaSourceEnumerator only supports adding new splits and not removing splits in split discovery.

See KafkaSourceBuilder for more details on how to configure this source.

See Also:

Serialized Form

Method Summary

Modifier and Type	Method and Description
static <OUT> KafkaSourceBuilder<OUT>	builder() Get a kafkaSourceBuilder to build a KafkaSource.
SplitEnumerator<KafkaPartitionSplit,KafkaSourceEnumState>	createEnumerator(SplitEnumeratorContext<KafkaPartitionSplit> enumContext) Creates a new SplitEnumerator for this source, starting a new input.
SourceReader<OUT,KafkaPartitionSplit>	createReader(SourceReaderContext readerContext) Creates a new reader to read data from the splits it gets assigned.
Boundedness	getBoundedness() Get the boundedness of this source.
SimpleVersionedSerializer<KafkaSourceEnumState>	getEnumeratorCheckpointSerializer() Creates the serializer for the SplitEnumerator checkpoint.
TypeInformation<OUT>	getProducedType() Gets the data type (as a TypeInformation) produced by this function or input format.
SimpleVersionedSerializer<KafkaPartitionSplit>	getSplitSerializer() Creates a serializer for the source splits.
SplitEnumerator<KafkaPartitionSplit,KafkaSourceEnumState>	restoreEnumerator(SplitEnumeratorContext<KafkaPartitionSplit> enumContext, KafkaSourceEnumState checkpoint) Restores an enumerator from a checkpoint.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

builder

public static <OUT> KafkaSourceBuilder<OUT> builder()

Get a kafkaSourceBuilder to build a KafkaSource.

Returns:

a Kafka source builder.

getBoundedness

public Boundedness getBoundedness()

Description copied from interface: Source

Get the boundedness of this source.

Specified by:

getBoundedness in interface Source<OUT,KafkaPartitionSplit,KafkaSourceEnumState>

Returns:

the boundedness of this source.

createReader

@Internal
public SourceReader<OUT,KafkaPartitionSplit> createReader(SourceReaderContext readerContext)
                                                             throws Exception

Description copied from interface: Source

Creates a new reader to read data from the splits it gets assigned. The reader starts fresh and does not have any state to resume.

Specified by:

createReader in interface Source<OUT,KafkaPartitionSplit,KafkaSourceEnumState>

Parameters:

readerContext - The context for the source reader.

Returns:

A new SourceReader.

Throws:

Exception - The implementor is free to forward all exceptions directly. Exceptions thrown from this method cause task failure/recovery.

createEnumerator

@Internal
public SplitEnumerator<KafkaPartitionSplit,KafkaSourceEnumState> createEnumerator(SplitEnumeratorContext<KafkaPartitionSplit> enumContext)

Description copied from interface: Source

Creates a new SplitEnumerator for this source, starting a new input.

Specified by:

createEnumerator in interface Source<OUT,KafkaPartitionSplit,KafkaSourceEnumState>

Parameters:

enumContext - The context for the split enumerator.

Returns:

A new SplitEnumerator.

restoreEnumerator

@Internal
public SplitEnumerator<KafkaPartitionSplit,KafkaSourceEnumState> restoreEnumerator(SplitEnumeratorContext<KafkaPartitionSplit> enumContext,
                                                                                             KafkaSourceEnumState checkpoint)
                                                                                      throws IOException

Description copied from interface: Source

Restores an enumerator from a checkpoint.

Specified by:

restoreEnumerator in interface Source<OUT,KafkaPartitionSplit,KafkaSourceEnumState>

Parameters:

enumContext - The context for the restored split enumerator.

checkpoint - The checkpoint to restore the SplitEnumerator from.

Returns:

A SplitEnumerator restored from the given checkpoint.

Throws:

IOException

getSplitSerializer

@Internal
public SimpleVersionedSerializer<KafkaPartitionSplit> getSplitSerializer()

Description copied from interface: Source

Creates a serializer for the source splits. Splits are serialized when sending them from enumerator to reader, and when checkpointing the reader's current state.

Specified by:

getSplitSerializer in interface Source<OUT,KafkaPartitionSplit,KafkaSourceEnumState>

Returns:

The serializer for the split type.

getEnumeratorCheckpointSerializer

@Internal
public SimpleVersionedSerializer<KafkaSourceEnumState> getEnumeratorCheckpointSerializer()

Description copied from interface: Source

Creates the serializer for the SplitEnumerator checkpoint. The serializer is used for the result of the SplitEnumerator.snapshotState(long) method.

Specified by:

getEnumeratorCheckpointSerializer in interface Source<OUT,KafkaPartitionSplit,KafkaSourceEnumState>

Returns:

The serializer for the SplitEnumerator checkpoint.

getProducedType

public TypeInformation<OUT> getProducedType()

Description copied from interface: ResultTypeQueryable

Gets the data type (as a TypeInformation) produced by this function or input format.

Specified by:

getProducedType in interface ResultTypeQueryable<OUT>

Returns:

The data type produced by this function or input format.