InputT
- Type of the elements handled by this sink@PublicEvolving public class KinesisStreamsSink<InputT> extends AsyncSinkBase<InputT,software.amazon.awssdk.services.kinesis.model.PutRecordsRequestEntry>
AsyncSinkBase
.
The sink internally uses a KinesisAsyncClient
to communicate with the AWS endpoint.
The behaviour of the buffering may be specified by providing configuration during the sink build time.
maxBatchSize
: the maximum size of a batch of entries that may be sent to KDS
maxInFlightRequests
: the maximum number of in flight requests that may exist, if
any more in flight requests need to be initiated once the maximum has been reached, then it
will be blocked until some have completed
maxBufferedRequests
: the maximum number of elements held in the buffer, requests to
add elements will be blocked while the number of elements in the buffer is at the maximum
maxBatchSizeInBytes
: the maximum size of a batch of entries that may be sent to KDS
measured in bytes
maxTimeInBufferMS
: the maximum amount of time an entry is allowed to live in the
buffer, if any element reaches this age, the entire buffer will be flushed immediately
maxRecordSizeInBytes
: the maximum size of a record the sink will accept into the
buffer, a record of size larger than this will be rejected when passed to the sink
failOnError
: when an exception is encountered while persisting to Kinesis Data
Streams, the job will fail immediately if failOnError is set
Please see the writer implementation in KinesisStreamsSinkWriter
StatefulSink.StatefulSinkWriter<InputT,WriterStateT>, StatefulSink.WithCompatibleState
Sink.InitContext
Modifier and Type | Method and Description |
---|---|
static <InputT> KinesisStreamsSinkBuilder<InputT> |
builder()
Create a
KinesisStreamsSinkBuilder to allow the fluent construction of a new KinesisStreamsSink . |
StatefulSink.StatefulSinkWriter<InputT,BufferedRequestState<software.amazon.awssdk.services.kinesis.model.PutRecordsRequestEntry>> |
createWriter(Sink.InitContext context)
Create a
StatefulSink.StatefulSinkWriter . |
SimpleVersionedSerializer<BufferedRequestState<software.amazon.awssdk.services.kinesis.model.PutRecordsRequestEntry>> |
getWriterStateSerializer()
Any stateful sink needs to provide this state serializer and implement
StatefulSink.StatefulSinkWriter.snapshotState(long) properly. |
StatefulSink.StatefulSinkWriter<InputT,BufferedRequestState<software.amazon.awssdk.services.kinesis.model.PutRecordsRequestEntry>> |
restoreWriter(Sink.InitContext context,
Collection<BufferedRequestState<software.amazon.awssdk.services.kinesis.model.PutRecordsRequestEntry>> recoveredState)
Create a
StatefulSink.StatefulSinkWriter from a recovered state. |
getElementConverter, getMaxBatchSize, getMaxBatchSizeInBytes, getMaxBufferedRequests, getMaxInFlightRequests, getMaxRecordSizeInBytes, getMaxTimeInBufferMS
public static <InputT> KinesisStreamsSinkBuilder<InputT> builder()
KinesisStreamsSinkBuilder
to allow the fluent construction of a new KinesisStreamsSink
.InputT
- type of incoming recordsKinesisStreamsSinkBuilder
@Internal public StatefulSink.StatefulSinkWriter<InputT,BufferedRequestState<software.amazon.awssdk.services.kinesis.model.PutRecordsRequestEntry>> createWriter(Sink.InitContext context) throws IOException
StatefulSink
StatefulSink.StatefulSinkWriter
.context
- the runtime context.IOException
- for any failure during creation.@Internal public SimpleVersionedSerializer<BufferedRequestState<software.amazon.awssdk.services.kinesis.model.PutRecordsRequestEntry>> getWriterStateSerializer()
StatefulSink
StatefulSink.StatefulSinkWriter.snapshotState(long)
properly. The respective state is used in #restoreWriter(InitContext, Collection)
on recovery.@Internal public StatefulSink.StatefulSinkWriter<InputT,BufferedRequestState<software.amazon.awssdk.services.kinesis.model.PutRecordsRequestEntry>> restoreWriter(Sink.InitContext context, Collection<BufferedRequestState<software.amazon.awssdk.services.kinesis.model.PutRecordsRequestEntry>> recoveredState) throws IOException
StatefulSink
StatefulSink.StatefulSinkWriter
from a recovered state.context
- the runtime context.IOException
- for any failure during creation.Copyright © 2014–2023 The Apache Software Foundation. All rights reserved.