public static class SourceReaderTestBase.ValidatingSourceOutput extends Object implements ReaderOutput<Integer>
| Constructor and Description |
|---|
ValidatingSourceOutput() |
| Modifier and Type | Method and Description |
|---|---|
void |
collect(Integer element)
Emit a record without a timestamp.
|
void |
collect(Integer element,
long timestamp)
Emit a record with a timestamp.
|
int |
count() |
SourceOutput<Integer> |
createOutputForSplit(String splitId)
Creates a
SourceOutput for a specific Source Split. |
void |
emitWatermark(Watermark watermark)
Emits the given watermark.
|
void |
markIdle()
Marks this output as idle, meaning that downstream operations do not wait for watermarks from
this output.
|
void |
releaseOutputForSplit(String splitId)
Releases the
SourceOutput created for the split with the given ID. |
void |
validate() |
public void collect(Integer element)
ReaderOutputUse this method if the source system does not have a notion of records with timestamps.
The events later pass through a TimestampAssigner, which attaches a timestamp to
the event based on the event's contents. For example a file source with JSON records would
not have a generic timestamp from the file reading and JSON parsing process, and thus use
this method to produce initially a record without a timestamp. The TimestampAssigner
in the next step would be used to extract timestamp from a field of the JSON object.
collect in interface ReaderOutput<Integer>collect in interface SourceOutput<Integer>element - the record to emit.public void collect(Integer element, long timestamp)
ReaderOutputUse this method if the source system has timestamps attached to records. Typical examples would be Logs, PubSubs, or Message Queues, like Kafka or Kinesis, which store a timestamp with each event.
The events typically still pass through a TimestampAssigner, which may decide to
either use this source-provided timestamp, or replace it with a timestamp stored within the
event (for example if the event was a JSON object one could configure aTimestampAssigner that
extracts one of the object's fields and uses that as a timestamp).
collect in interface ReaderOutput<Integer>collect in interface SourceOutput<Integer>element - the record to emit.timestamp - the timestamp of the record.public void validate()
public int count()
public void emitWatermark(Watermark watermark)
ReaderOutputEmitting a watermark also implicitly marks the stream as active, ending previously marked idleness.
emitWatermark in interface WatermarkOutputemitWatermark in interface ReaderOutput<Integer>public void markIdle()
ReaderOutputAn output becomes active again as soon as the next watermark is emitted.
markIdle in interface WatermarkOutputmarkIdle in interface ReaderOutput<Integer>public SourceOutput<Integer> createOutputForSplit(String splitId)
ReaderOutputSourceOutput for a specific Source Split. Use these outputs if you want to
run split-local logic, like watermark generation.
If a split-local output was already created for this split-ID, the method will return that instance, so that only one split-local output exists per split-ID.
IMPORTANT: After the split has been finished, it is crucial to release the created output again. Otherwise it will continue to contribute to the watermark generation like a perpetually stalling source split, and may hold back the watermark indefinitely.
createOutputForSplit in interface ReaderOutput<Integer>ReaderOutput.releaseOutputForSplit(String)public void releaseOutputForSplit(String splitId)
ReaderOutputSourceOutput created for the split with the given ID.releaseOutputForSplit in interface ReaderOutput<Integer>ReaderOutput.createOutputForSplit(String)Copyright © 2014–2021 The Apache Software Foundation. All rights reserved.