OUT
- the output type of the source.@PublicEvolving public class KafkaSource<OUT> extends Object implements Source<OUT,KafkaPartitionSplit,KafkaSourceEnumState>, ResultTypeQueryable<OUT>
KafkaSourceBuilder
to construct a KafkaSource
. The following example shows how to create a KafkaSource emitting records of
String
type.
KafkaSource<String> source = KafkaSource
.<String>builder()
.setBootstrapServers(KafkaSourceTestEnv.brokerConnectionStrings)
.setGroupId("MyGroup")
.setTopics(Arrays.asList(TOPIC1, TOPIC2))
.setDeserializer(new TestingKafkaRecordDeserializationSchema())
.setStartingOffsets(OffsetsInitializer.earliest())
.build();
KafkaSourceEnumerator
only supports
adding new splits and not removing splits in split discovery.
See KafkaSourceBuilder
for more details on how to configure this source.
public static <OUT> KafkaSourceBuilder<OUT> builder()
KafkaSource
.public Boundedness getBoundedness()
Source
getBoundedness
in interface Source<OUT,KafkaPartitionSplit,KafkaSourceEnumState>
@Internal public SourceReader<OUT,KafkaPartitionSplit> createReader(SourceReaderContext readerContext) throws Exception
SourceReaderFactory
createReader
in interface SourceReaderFactory<OUT,KafkaPartitionSplit>
readerContext
- The context
for the source reader.Exception
- The implementor is free to forward all exceptions directly. Exceptions
thrown from this method cause task failure/recovery.@Internal public SplitEnumerator<KafkaPartitionSplit,KafkaSourceEnumState> createEnumerator(SplitEnumeratorContext<KafkaPartitionSplit> enumContext)
Source
createEnumerator
in interface Source<OUT,KafkaPartitionSplit,KafkaSourceEnumState>
enumContext
- The context
for the split enumerator.@Internal public SplitEnumerator<KafkaPartitionSplit,KafkaSourceEnumState> restoreEnumerator(SplitEnumeratorContext<KafkaPartitionSplit> enumContext, KafkaSourceEnumState checkpoint) throws IOException
Source
restoreEnumerator
in interface Source<OUT,KafkaPartitionSplit,KafkaSourceEnumState>
enumContext
- The context
for the restored split
enumerator.checkpoint
- The checkpoint to restore the SplitEnumerator from.IOException
@Internal public SimpleVersionedSerializer<KafkaPartitionSplit> getSplitSerializer()
Source
getSplitSerializer
in interface Source<OUT,KafkaPartitionSplit,KafkaSourceEnumState>
@Internal public SimpleVersionedSerializer<KafkaSourceEnumState> getEnumeratorCheckpointSerializer()
Source
SplitEnumerator
checkpoint. The serializer is used for
the result of the SplitEnumerator.snapshotState(long)
method.getEnumeratorCheckpointSerializer
in interface Source<OUT,KafkaPartitionSplit,KafkaSourceEnumState>
public TypeInformation<OUT> getProducedType()
ResultTypeQueryable
TypeInformation
) produced by this function or input format.getProducedType
in interface ResultTypeQueryable<OUT>
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.