org.apache.flink.streaming.connectors.kinesis.proxy

Interface KinesisProxyInterface

All Known Implementing Classes:

DynamoDBStreamsProxy, KinesisProxy

@Internal
public interface KinesisProxyInterface

Interface for a Kinesis proxy that operates on multiple Kinesis streams within the same AWS service region.

Method Summary

Modifier and Type	Method and Description
com.amazonaws.services.kinesis.model.GetRecordsResult	getRecords(String shardIterator, int maxRecordsToGet) Get the next batch of data records using a specific shard iterator.
String	getShardIterator(StreamShardHandle shard, String shardIteratorType, Object startingMarker) Get a shard iterator from the specified position in a shard.
GetShardListResult	getShardList(Map<String,String> streamNamesWithLastSeenShardIds) Get shard list of multiple Kinesis streams, ignoring the shards of each stream before a specified last seen shard id.

Method Detail

getShardIterator

String getShardIterator(StreamShardHandle shard,
                        String shardIteratorType,
                        Object startingMarker)
                 throws InterruptedException

Get a shard iterator from the specified position in a shard. The retrieved shard iterator can be used in getRecords(String, int)} to read data from the Kinesis shard.

Parameters:

shard - the shard to get the iterator

shardIteratorType - the iterator type, defining how the shard is to be iterated (one of: TRIM_HORIZON, LATEST, AT_TIMESTAMP, AT_SEQUENCE_NUMBER, AFTER_SEQUENCE_NUMBER)

startingMarker - should be null if shardIteratorType is TRIM_HORIZON or LATEST, should be a Date value if shardIteratorType is AT_TIMESTAMP, should be a String representing the sequence number if shardIteratorType is AT_SEQUENCE_NUMBER, AFTER_SEQUENCE_NUMBER

Returns:

shard iterator which can be used to read data from Kinesis

Throws:

InterruptedException - this method will retry with backoff if AWS Kinesis complains that the operation has exceeded the rate limit; this exception will be thrown if the backoff is interrupted.

getRecords

com.amazonaws.services.kinesis.model.GetRecordsResult getRecords(String shardIterator,
                                                                 int maxRecordsToGet)
                                                          throws InterruptedException

Get the next batch of data records using a specific shard iterator.

Parameters:

shardIterator - a shard iterator that encodes info about which shard to read and where to start reading

maxRecordsToGet - the maximum amount of records to retrieve for this batch

Returns:

the batch of retrieved records, also with a shard iterator that can be used to get the next batch

Throws:

InterruptedException - this method will retry with backoff if AWS Kinesis complains that the operation has exceeded the rate limit; this exception will be thrown if the backoff is interrupted.

getShardList

GetShardListResult getShardList(Map<String,String> streamNamesWithLastSeenShardIds)
                         throws InterruptedException

Get shard list of multiple Kinesis streams, ignoring the shards of each stream before a specified last seen shard id.

Parameters:

streamNamesWithLastSeenShardIds - a map with stream as key, and last seen shard id as value

Returns:

result of the shard list query

Throws:

InterruptedException - this method will retry with backoff if AWS Kinesis complains that the operation has exceeded the rate limit; this exception will be thrown if the backoff is interrupted.