Constructor and Description |
---|
LegacyFetcher(Map<KafkaTopicPartition,Long> initialPartitionsToRead,
Properties props,
String taskName,
ClassLoader userCodeClassloader)
Create a LegacyFetcher instance.
|
Modifier and Type | Method and Description |
---|---|
void |
close()
Closes the fetcher.
|
static List<String> |
getTopics(List<org.apache.flink.streaming.connectors.kafka.internals.LegacyFetcher.FetchPartition> partitionsList)
Returns a unique list of topics from the topic partition list
|
<T> void |
run(SourceFunction.SourceContext<T> sourceContext,
KeyedDeserializationSchema<T> deserializer,
HashMap<KafkaTopicPartition,Long> lastOffsets)
Starts fetch data from Kafka and emitting it into the stream.
|
void |
stopWithError(Throwable error)
Reports an error from a fetch thread.
|
public LegacyFetcher(Map<KafkaTopicPartition,Long> initialPartitionsToRead, Properties props, String taskName, ClassLoader userCodeClassloader)
initialPartitionsToRead
- Map of partitions to read. The offset passed is the last-fetched-offset (not the next-offset-to-fetch).props
- kafka propertiestaskName
- name of the parent taskuserCodeClassloader
- classloader for loading user codepublic void close()
Fetcher
Fetcher.run(SourceFunction.SourceContext, KeyedDeserializationSchema, HashMap)
method and eventually
close underlying connections and release all resources.public <T> void run(SourceFunction.SourceContext<T> sourceContext, KeyedDeserializationSchema<T> deserializer, HashMap<KafkaTopicPartition,Long> lastOffsets) throws Exception
Fetcher
To provide exactly once guarantees, the fetcher needs emit a record and update the update of the last consumed offset in one atomic operation:
while (running) {
T next = ...
long offset = ...
int partition = ...
synchronized (sourceContext.getCheckpointLock()) {
sourceContext.collect(next);
lastOffsets[partition] = offset;
}
}
run
in interface Fetcher
T
- The type of elements produced by the fetcher and emitted to the source context.sourceContext
- The source context to emit elements to.deserializer
- The deserializer to decode the raw values with.lastOffsets
- The map into which to store the offsets for which elements are emitted (operator state)Exception
public void stopWithError(Throwable error)
stopWithError
in interface Fetcher
error
- The error to report.public static List<String> getTopics(List<org.apache.flink.streaming.connectors.kafka.internals.LegacyFetcher.FetchPartition> partitionsList)
partitionsList
- A lost of FetchPartitions'sCopyright © 2014–2017 The Apache Software Foundation. All rights reserved.