org.apache.flink.runtime.io.network.api.writer

Class RecordWriter<T extends IOReadableWritable>

java.lang.Object

org.apache.flink.runtime.io.network.api.writer.RecordWriter<T>

Type Parameters:

T - the type of the record that can be emitted with this record writer

All Implemented Interfaces:

AvailabilityProvider

Direct Known Subclasses:

BroadcastRecordWriter, ChannelSelectorRecordWriter

public abstract class RecordWriter<T extends IOReadableWritable>
extends Object
implements AvailabilityProvider

An abstract record-oriented runtime result writer.

The RecordWriter wraps the runtime's ResultPartitionWriter and takes care of serializing records into buffers.

Important: it is necessary to call flushAll() after all records have been written with emit(IOReadableWritable). This ensures that all produced records are written to the output stream (incl. partially filled ones).

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

AvailabilityProvider.AvailabilityHelper

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEFAULT_OUTPUT_FLUSH_THREAD_NAME Default name for the output flush thread, if no name with a task reference is given.
protected Meter	idleTimeMsPerSecond
protected int	numberOfChannels
protected Random	rng
protected RecordSerializer<T>	serializer

Fields inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

AVAILABLE

Method Summary

Modifier and Type	Method and Description
protected void	addBufferConsumer(BufferConsumer consumer, int targetChannel)
abstract void	broadcastEmit(T record) This is used to broadcast streaming Watermarks in-band with records.
void	broadcastEvent(AbstractEvent event)
void	broadcastEvent(AbstractEvent event, boolean isPriorityEvent)
protected void	checkErroneous()
abstract void	clearBuffers() Closes the BufferBuilders for all the channels.
void	close() Closes the writer.
protected boolean	copyFromSerializerToTargetChannel(int targetChannel)
abstract void	emit(T record) This is used to send regular records.
protected void	emit(T record, int targetChannel)
protected void	finishBufferBuilder(BufferBuilder bufferBuilder)
void	flushAll()
protected void	flushTargetPartition(int targetChannel)
CompletableFuture<?>	getAvailableFuture()
Meter	getIdleTimeMsPerSecond()
abstract void	randomEmit(T record) This is used to send LatencyMarks to a random target channel.
BufferBuilder	requestNewBufferBuilder(int targetChannel) Requests a new BufferBuilder for the target channel and returns it.
void	setMetricGroup(TaskIOMetricGroup metrics) Sets the metric group for this RecordWriter.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

isApproximatelyAvailable, isAvailable

Field Detail

DEFAULT_OUTPUT_FLUSH_THREAD_NAME

@VisibleForTesting
public static final String DEFAULT_OUTPUT_FLUSH_THREAD_NAME

Default name for the output flush thread, if no name with a task reference is given.

See Also:

Constant Field Values

numberOfChannels

protected final int numberOfChannels

serializer

protected final RecordSerializer<T extends IOReadableWritable> serializer

rng

protected final Random rng

idleTimeMsPerSecond

protected Meter idleTimeMsPerSecond

Method Detail

emit

protected void emit(T record,
                    int targetChannel)
             throws IOException,
                    InterruptedException

Throws:

IOException

InterruptedException

copyFromSerializerToTargetChannel

protected boolean copyFromSerializerToTargetChannel(int targetChannel)
                                             throws IOException,
                                                    InterruptedException

Parameters:

targetChannel -

Returns:

true if the intermediate serialization buffer should be pruned

Throws:

IOException

InterruptedException

broadcastEvent

public void broadcastEvent(AbstractEvent event)
                    throws IOException

Throws:

IOException

broadcastEvent

public void broadcastEvent(AbstractEvent event,
                           boolean isPriorityEvent)
                    throws IOException

Throws:

IOException

flushAll

public void flushAll()

flushTargetPartition

protected void flushTargetPartition(int targetChannel)

setMetricGroup

public void setMetricGroup(TaskIOMetricGroup metrics)

Sets the metric group for this RecordWriter.

finishBufferBuilder

protected void finishBufferBuilder(BufferBuilder bufferBuilder)

getAvailableFuture

public CompletableFuture<?> getAvailableFuture()

Specified by:

getAvailableFuture in interface AvailabilityProvider

Returns:

a future that is completed if the respective provider is available.

emit

public abstract void emit(T record)
                   throws IOException,
                          InterruptedException

This is used to send regular records.

Throws:

IOException

InterruptedException

randomEmit

public abstract void randomEmit(T record)
                         throws IOException,
                                InterruptedException

This is used to send LatencyMarks to a random target channel.

Throws:

IOException

InterruptedException

broadcastEmit

public abstract void broadcastEmit(T record)
                            throws IOException,
                                   InterruptedException

This is used to broadcast streaming Watermarks in-band with records.

Throws:

IOException

InterruptedException

clearBuffers

public abstract void clearBuffers()

Closes the BufferBuilders for all the channels.

close

public void close()

Closes the writer. This stops the flushing thread (if there is one).

checkErroneous

protected void checkErroneous()
                       throws IOException

Throws:

IOException

addBufferConsumer

protected void addBufferConsumer(BufferConsumer consumer,
                                 int targetChannel)
                          throws IOException

Throws:

IOException

requestNewBufferBuilder

public BufferBuilder requestNewBufferBuilder(int targetChannel)
                                      throws IOException,
                                             InterruptedException

Requests a new BufferBuilder for the target channel and returns it.

Throws:

IOException

InterruptedException

getIdleTimeMsPerSecond

@VisibleForTesting
public Meter getIdleTimeMsPerSecond()