Package org.apache.flink.connector.testutils.source.reader

Class SourceReaderTestBase.ValidatingSourceOutput

java.lang.Object

org.apache.flink.connector.testutils.source.reader.SourceReaderTestBase.ValidatingSourceOutput

All Implemented Interfaces:

WatermarkOutput, ReaderOutput<Integer>, SourceOutput<Integer>

Enclosing class:

SourceReaderTestBase<SplitT extends SourceSplit>

public class SourceReaderTestBase.ValidatingSourceOutput
extends Object
implements ReaderOutput<Integer>

A source output that validates the output.

Constructor Summary

Constructors

Constructor	Description
ValidatingSourceOutput()

Method Summary

Modifier and Type	Method	Description
void	collect(Integer element)	Emit a record without a timestamp.
void	collect(Integer element, long timestamp)	Emit a record with a timestamp.
int	count()
SourceOutput<Integer>	createOutputForSplit(String splitId)	Creates a SourceOutput for a specific Source Split.
void	emitWatermark(Watermark watermark)	Emits the given watermark.
void	markActive()	Marks this output as active, meaning that downstream operations should wait for watermarks from this output.
void	markIdle()	Marks this output as idle, meaning that downstream operations do not wait for watermarks from this output.
void	releaseOutputForSplit(String splitId)	Releases the SourceOutput created for the split with the given ID.
void	validate()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ValidatingSourceOutput

public ValidatingSourceOutput()

Method Detail

collect

public void collect(Integer element)

Description copied from interface: ReaderOutput

Emit a record without a timestamp.

Use this method if the source system does not have a notion of records with timestamps.

The events later pass through a TimestampAssigner, which attaches a timestamp to the event based on the event's contents. For example a file source with JSON records would not have a generic timestamp from the file reading and JSON parsing process, and thus use this method to produce initially a record without a timestamp. The TimestampAssigner in the next step would be used to extract timestamp from a field of the JSON object.

Specified by:

collect in interface ReaderOutput<Integer>

Specified by:

collect in interface SourceOutput<Integer>

Parameters:

element - the record to emit.

collect

public void collect(Integer element,
                    long timestamp)

Description copied from interface: ReaderOutput

Emit a record with a timestamp.

Use this method if the source system has timestamps attached to records. Typical examples would be Logs, PubSubs, or Message Queues, like Kafka or Kinesis, which store a timestamp with each event.

The events typically still pass through a TimestampAssigner, which may decide to either use this source-provided timestamp, or replace it with a timestamp stored within the event (for example if the event was a JSON object one could configure aTimestampAssigner that extracts one of the object's fields and uses that as a timestamp).

Specified by:

collect in interface ReaderOutput<Integer>

Specified by:

collect in interface SourceOutput<Integer>

Parameters:

element - the record to emit.

timestamp - the timestamp of the record.

validate

public void validate()

count

public int count()

emitWatermark

public void emitWatermark(Watermark watermark)

Description copied from interface: ReaderOutput

Emits the given watermark.

Emitting a watermark also implicitly marks the stream as active, ending previously marked idleness.

Specified by:

emitWatermark in interface ReaderOutput<Integer>

Specified by:

emitWatermark in interface WatermarkOutput

markIdle

public void markIdle()

Description copied from interface: ReaderOutput

Marks this output as idle, meaning that downstream operations do not wait for watermarks from this output.

An output becomes active again as soon as the next watermark is emitted.

Specified by:

markIdle in interface ReaderOutput<Integer>

Specified by:

markIdle in interface WatermarkOutput

markActive

public void markActive()

Description copied from interface: WatermarkOutput

Marks this output as active, meaning that downstream operations should wait for watermarks from this output.

Specified by:

markActive in interface WatermarkOutput

createOutputForSplit

public SourceOutput<Integer> createOutputForSplit(String splitId)

Description copied from interface: ReaderOutput

Creates a SourceOutput for a specific Source Split. Use these outputs if you want to run split-local logic, like watermark generation.

If a split-local output was already created for this split-ID, the method will return that instance, so that only one split-local output exists per split-ID.

IMPORTANT: After the split has been finished, it is crucial to release the created output again. Otherwise it will continue to contribute to the watermark generation like a perpetually stalling source split, and may hold back the watermark indefinitely.

Specified by:

createOutputForSplit in interface ReaderOutput<Integer>

See Also:

ReaderOutput.releaseOutputForSplit(String)

releaseOutputForSplit

public void releaseOutputForSplit(String splitId)

Description copied from interface: ReaderOutput

Releases the SourceOutput created for the split with the given ID.

Specified by:

releaseOutputForSplit in interface ReaderOutput<Integer>

See Also:

ReaderOutput.createOutputForSplit(String)