T
- @Internal public abstract class RecordEmitter<T extends TimestampedValue> extends Object implements Runnable
Records are organized into per producer queues that will block when capacity is exhausted.
Records are emitted by selecting the oldest available element of all producer queues, as long as the timestamp does not exceed the current shared watermark plus allowed lookahead interval.
Modifier and Type | Class and Description |
---|---|
static interface |
RecordEmitter.RecordQueue<T>
Accepts records from readers.
|
Modifier and Type | Field and Description |
---|---|
static int |
DEFAULT_QUEUE_CAPACITY
The default capacity of a single queue.
|
Constructor and Description |
---|
RecordEmitter(int queueCapacity) |
Modifier and Type | Method and Description |
---|---|
protected abstract void |
emit(T record,
RecordEmitter.RecordQueue<T> source)
Emit the record.
|
protected void |
emitRecords() |
RecordEmitter.RecordQueue<T> |
getQueue(int producerIndex)
The queue for the given producer (i.e.
|
String |
printInfo()
Summary of emit queues that can be used for logging.
|
void |
run()
Run loop, does not return unless
stop() was called. |
void |
setCurrentWatermark(long watermark)
Set the current watermark.
|
void |
setMaxLookaheadMillis(long maxLookaheadMillis)
How far ahead of the watermark records should be emitted.
|
void |
stop() |
public static final int DEFAULT_QUEUE_CAPACITY
Larger queue size can lead to higher throughput, but also to very high heap space consumption, depending on the size of elements.
Note that this is difficult to tune, because it does not take into account the size of individual objects.
public RecordEmitter.RecordQueue<T> getQueue(int producerIndex)
The producer may hold on to the queue for subsequent records.
producerIndex
- public void setMaxLookaheadMillis(long maxLookaheadMillis)
Needs to account for the latency of obtaining the global watermark.
maxLookaheadMillis
- public void setCurrentWatermark(long watermark)
This watermark will be used to control which records to emit from the queues of pending
elements. When an element timestamp is ahead of the watermark by at least maxLookaheadMillis
, elements in that queue will wait until the watermark advances.
watermark
- public void run()
stop()
was called.public void stop()
protected void emitRecords()
protected abstract void emit(T record, RecordEmitter.RecordQueue<T> source)
public String printInfo()
Copyright © 2014–2022 The Apache Software Foundation. All rights reserved.