org.apache.flink.streaming.connectors.kinesis.util

Class KinesisConfigUtil

java.lang.Object

org.apache.flink.streaming.connectors.kinesis.util.KinesisConfigUtil

@Internal
public class KinesisConfigUtil
extends Object

Utilities for Flink Kinesis connector configuration.

Field Summary

Fields

Modifier and Type	Field and Description
protected static String	AGGREGATION_MAX_COUNT Maximum number of items to pack into an aggregated record.
protected static String	COLLECTION_MAX_COUNT Maximum number of items to pack into an PutRecords request.
protected static long	DEFAULT_RATE_LIMIT Default values for RateLimit.
protected static int	DEFAULT_THREAD_POOL_SIZE Default values for ThreadPoolSize.
protected static com.amazonaws.services.kinesis.producer.KinesisProducerConfiguration.ThreadingModel	DEFAULT_THREADING_MODEL Default value for ThreadingModel.
protected static String	RATE_LIMIT Limits the maximum allowed put rate for a shard, as a percentage of the backend limits.
protected static String	THREAD_POOL_SIZE The maximum number of threads that the native process' thread pool will be configured with.
protected static String	THREADING_MODEL The threading model that KinesisProducer will use.

Constructor Summary

Constructors

Constructor and Description
KinesisConfigUtil()

Method Summary

Modifier and Type	Method and Description
static Properties	backfillConsumerKeys(Properties configProps) A set of configuration paremeters associated with the describeStreams API may be used if: 1) an legacy client wants to consume from Kinesis 2) a current client wants to consumer from DynamoDB streams
static com.amazonaws.services.kinesis.producer.KinesisProducerConfiguration	getValidatedProducerConfiguration(Properties config) Validate configuration properties for FlinkKinesisProducer, and return a constructed KinesisProducerConfiguration.
static Properties	replaceDeprecatedProducerKeys(Properties configProps) Replace deprecated configuration properties for FlinkKinesisProducer.
static void	validateAwsConfiguration(Properties config) Validate configuration properties related to Amazon AWS service.
static void	validateConsumerConfiguration(Properties config) Validate configuration properties for FlinkKinesisConsumer.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

COLLECTION_MAX_COUNT

protected static final String COLLECTION_MAX_COUNT

Maximum number of items to pack into an PutRecords request. *

See Also:

Constant Field Values

AGGREGATION_MAX_COUNT

protected static final String AGGREGATION_MAX_COUNT

Maximum number of items to pack into an aggregated record. *

See Also:

Constant Field Values

RATE_LIMIT

protected static final String RATE_LIMIT

Limits the maximum allowed put rate for a shard, as a percentage of the backend limits. The default value is set as 100% in Flink. KPL's default value is 150% but it makes KPL throw RateLimitExceededException too frequently and breaks Flink sink as a result.

See Also:

Constant Field Values

THREADING_MODEL

protected static final String THREADING_MODEL

The threading model that KinesisProducer will use.

See Also:

Constant Field Values

THREAD_POOL_SIZE

protected static final String THREAD_POOL_SIZE

The maximum number of threads that the native process' thread pool will be configured with.

See Also:

Constant Field Values

DEFAULT_RATE_LIMIT

protected static final long DEFAULT_RATE_LIMIT

Default values for RateLimit. *

See Also:

Constant Field Values

DEFAULT_THREADING_MODEL

protected static final com.amazonaws.services.kinesis.producer.KinesisProducerConfiguration.ThreadingModel DEFAULT_THREADING_MODEL

Default value for ThreadingModel. *

DEFAULT_THREAD_POOL_SIZE

protected static final int DEFAULT_THREAD_POOL_SIZE

Default values for ThreadPoolSize. *

See Also:

Constant Field Values

Constructor Detail

KinesisConfigUtil

public KinesisConfigUtil()

Method Detail

validateConsumerConfiguration

public static void validateConsumerConfiguration(Properties config)

Validate configuration properties for FlinkKinesisConsumer.

replaceDeprecatedProducerKeys

public static Properties replaceDeprecatedProducerKeys(Properties configProps)

Replace deprecated configuration properties for FlinkKinesisProducer. This should be remove along with deprecated keys

backfillConsumerKeys

public static Properties backfillConsumerKeys(Properties configProps)

A set of configuration paremeters associated with the describeStreams API may be used if: 1) an legacy client wants to consume from Kinesis 2) a current client wants to consumer from DynamoDB streams

In the context of 1), the set of configurations needs to be translated to the corresponding configurations in the Kinesis listShards API. In the mean time, keep these configs since they are applicable in the context of 2), i.e., polling data from a DynamoDB stream.

Parameters:

configProps - original config properties.

Returns:

backfilled config properties.

getValidatedProducerConfiguration

public static com.amazonaws.services.kinesis.producer.KinesisProducerConfiguration getValidatedProducerConfiguration(Properties config)

Validate configuration properties for FlinkKinesisProducer, and return a constructed KinesisProducerConfiguration.

validateAwsConfiguration

public static void validateAwsConfiguration(Properties config)

Validate configuration properties related to Amazon AWS service.