public interface Fetcher
Modifier and Type | Method and Description |
---|---|
void |
close()
Closes the fetcher.
|
<T> void |
run(SourceFunction.SourceContext<T> sourceContext,
KeyedDeserializationSchema<T> valueDeserializer,
HashMap<KafkaTopicPartition,Long> lastOffsets)
Starts fetch data from Kafka and emitting it into the stream.
|
void |
stopWithError(Throwable t)
Exit run loop with given error and release all resources.
|
void close() throws IOException
run(SourceFunction.SourceContext, KeyedDeserializationSchema, HashMap)
method and eventually
close underlying connections and release all resources.IOException
<T> void run(SourceFunction.SourceContext<T> sourceContext, KeyedDeserializationSchema<T> valueDeserializer, HashMap<KafkaTopicPartition,Long> lastOffsets) throws Exception
To provide exactly once guarantees, the fetcher needs emit a record and update the update of the last consumed offset in one atomic operation:
while (running) {
T next = ...
long offset = ...
int partition = ...
synchronized (sourceContext.getCheckpointLock()) {
sourceContext.collect(next);
lastOffsets[partition] = offset;
}
}
T
- The type of elements produced by the fetcher and emitted to the source context.sourceContext
- The source context to emit elements to.valueDeserializer
- The deserializer to decode the raw values with.lastOffsets
- The map into which to store the offsets for which elements are emitted (operator state)Exception
void stopWithError(Throwable t)
t
- Error causeCopyright © 2014–2017 The Apache Software Foundation. All rights reserved.