This documentation is for an unreleased version of Apache Flink Stateful Functions. We recommend you use the latest stable version.

AWS Kinesis #

Stateful Functions offers an AWS Kinesis I/O Module for reading from and writing to Kinesis streams. It is based on Apache Flink’s Kinesis connector. Kinesis is configured in the module specification of your application.

Kinesis Ingress Spec #

A Kinesis ingress defines an input point that reads records from one or more streams.

kind: io.statefun.kinesis.v1/ingress
spec:
  id: com.example/users
  awsRegion:
    type: specific
    id: eu-west-1
  startupPosition:
    type: latest
  streams:
    - stream: user-stream
      valueType: com.example/User
      targets:
        - com.example.fn/greeter
  clientConfigProperties:
    - SocketTimeout: 9999
    - MaxConnections: 15

Please refer to the Kinesis client configuration documentation for the full list of available properties. Note that configuration passed using named methods will have higher precedence and overwrite their respective settings in the provided properties.

Startup Position #

The ingress allows configuring the startup position to be one of the following:

Latest (default) #

Start consuming from the latest position, i.e. head of the stream shards.

startupPosition:
  type: latest

Earliest #

Start consuming from the earliest position possible.

startupPosition:
  type: earliest

Date #

Starts from offsets that have an ingestion time larger than or equal to a specified date.

startupPosition:
  type: date
  date: 2020-02-01 04:15:00.00 Z

Kinesis Egress Spec #

A Kinesis egress defines an input point where functions can write out records to one or more streams.

kind: io.statefun.kinesis.v1/egress
spec:
  id: com.example/out
  awsRegion:
    type: specific
    id: eu-west-1
  awsCredentials:
    type: default
  maxOutstandingRecords: 9999
  clientConfigProperties:
    - ThreadingModel: POOLED
    - ThreadPoolSize: 10

Please refer to the Kinesis producer default configuration properties documentation for the full list of available properties.

AWS Region #

Both the Kinesis ingress and egress can be configured to a specific AWS region.

Default Provider Chain (default) #

Consults AWS’s default provider chain to determine the AWS region.

awsCredentials:
  type: default

Specific #

Specifies an AWS region using the region’s unique id.

awsCredentials:
  type: specific
  id: us-west-1

Custom Endpoint #

Connects to an AWS region through a non-standard AWS service endpoint. This is typically used only for development and testing purposes.

awsCredentials:
  type: custom-endpoint
  endpoint: https://localhost:4567
  id: us-west-1

AWS Credentials #

Both the Kinesis ingress and egress can be configured using standard AWS credential providers.

Default Provider Chain (default) #

Consults AWS’s default provider chain to determine the AWS credentials.

awsCredentials:
  type: default

Basic #

Specifies the AWS credentials directly with provided access key ID and secret access key strings.

awsCredentials:
  type: basic
  accessKeyId: access-key-id
  secretAccessKey: secret-access-key

Profile #

Specifies the AWS credentials using an AWS configuration profile, along with the profile’s configuration path.

awsCredentials:
  type: basic
  profileName: profile-name
  profilePath: /path/to/profile/config

Writing To Kinesis #

Functions write directly to Kinesis from their SDK context. See SDK specific documentation for more details.