OUT
- Data type to produce into Kinesis Streams@PublicEvolving public class FlinkKinesisProducer<OUT> extends RichSinkFunction<OUT> implements CheckpointedFunction
SinkFunction.Context
Modifier and Type | Field and Description |
---|---|
static String |
KINESIS_PRODUCER_METRIC_GROUP |
static String |
KINESIS_PRODUCER_RELEASE_HOOK_NAME |
static String |
METRIC_BACKPRESSURE_CYCLES |
static String |
METRIC_OUTSTANDING_RECORDS_COUNT |
Constructor and Description |
---|
FlinkKinesisProducer(KinesisSerializationSchema<OUT> schema,
Properties configProps)
Create a new FlinkKinesisProducer.
|
FlinkKinesisProducer(SerializationSchema<OUT> schema,
Properties configProps)
Create a new FlinkKinesisProducer.
|
Modifier and Type | Method and Description |
---|---|
void |
close()
Tear-down method for the user code.
|
protected com.amazonaws.services.kinesis.producer.KinesisProducer |
getKinesisProducer(com.amazonaws.services.kinesis.producer.KinesisProducerConfiguration producerConfig)
Creates a
KinesisProducer . |
void |
initializeState(FunctionInitializationContext context)
This method is called when the parallel function instance is created during distributed
execution.
|
void |
invoke(OUT value,
SinkFunction.Context context)
Writes the given value to the sink.
|
void |
open(Configuration parameters)
Initialization method for the function.
|
void |
setCustomPartitioner(KinesisPartitioner<OUT> partitioner) |
void |
setDefaultPartition(String defaultPartition)
Set default partition id.
|
void |
setDefaultStream(String defaultStream)
Set a default stream name.
|
void |
setFailOnError(boolean failOnError)
If set to true, the producer will immediately fail with an exception on any error.
|
void |
setQueueLimit(int queueLimit)
The
KinesisProducer holds an unbounded queue internally. |
void |
snapshotState(FunctionSnapshotContext context)
This method is called when a snapshot for a checkpoint is requested.
|
getIterationRuntimeContext, getRuntimeContext, setRuntimeContext
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
invoke
public static final String KINESIS_PRODUCER_METRIC_GROUP
public static final String METRIC_BACKPRESSURE_CYCLES
public static final String METRIC_OUTSTANDING_RECORDS_COUNT
public static final String KINESIS_PRODUCER_RELEASE_HOOK_NAME
public FlinkKinesisProducer(SerializationSchema<OUT> schema, Properties configProps)
SerializationSchema
.schema
- Serialization schema for the data typeconfigProps
- The properties used to configure KinesisProducer, including AWS
credentials and AWS regionpublic FlinkKinesisProducer(KinesisSerializationSchema<OUT> schema, Properties configProps)
KinesisSerializationSchema
.schema
- Kinesis serialization schema for the data typeconfigProps
- The properties used to configure KinesisProducer, including AWS
credentials and AWS regionpublic void setFailOnError(boolean failOnError)
failOnError
- Error behavior flagpublic void setQueueLimit(int queueLimit)
KinesisProducer
holds an unbounded queue internally. To avoid memory problems
under high loads, a limit can be employed above which the internal queue will be flushed,
thereby applying backpressure.queueLimit
- The maximum length of the internal queue before backpressuringpublic void setDefaultStream(String defaultStream)
defaultStream
- Name of the default Kinesis streampublic void setDefaultPartition(String defaultPartition)
defaultPartition
- Name of the default partitionpublic void setCustomPartitioner(KinesisPartitioner<OUT> partitioner)
public void open(Configuration parameters) throws Exception
RichFunction
The configuration object passed to the function can be used for configuration and initialization. The configuration contains all parameters that were configured on the function in the program composition.
public class MyFilter extends RichFilterFunction<String> {
private String searchString;
public void open(Configuration parameters) {
this.searchString = parameters.getString("foo");
}
public boolean filter(String value) {
return value.equals(searchString);
}
}
By default, this method does nothing.
open
in interface RichFunction
open
in class AbstractRichFunction
parameters
- The configuration containing the parameters attached to the contract.Exception
- Implementations may forward exceptions, which are caught by the runtime.
When the runtime catches an exception, it aborts the task and lets the fail-over logic
decide whether to retry the task execution.Configuration
public void invoke(OUT value, SinkFunction.Context context) throws Exception
SinkFunction
You have to override this method when implementing a SinkFunction
, this is a
default
method for backward compatibility with the old-style method only.
invoke
in interface SinkFunction<OUT>
value
- The input record.context
- Additional context about the input record.Exception
- This method may throw exceptions. Throwing an exception will cause the
operation to fail and may trigger recovery.public void close() throws Exception
RichFunction
This method can be used for clean up work.
close
in interface RichFunction
close
in class AbstractRichFunction
Exception
- Implementations may forward exceptions, which are caught by the runtime.
When the runtime catches an exception, it aborts the task and lets the fail-over logic
decide whether to retry the task execution.public void initializeState(FunctionInitializationContext context) throws Exception
CheckpointedFunction
initializeState
in interface CheckpointedFunction
context
- the context for initializing the operatorException
- Thrown, if state could not be created ot restored.public void snapshotState(FunctionSnapshotContext context) throws Exception
CheckpointedFunction
FunctionInitializationContext
when the Function was initialized, or offered now by FunctionSnapshotContext
itself.snapshotState
in interface CheckpointedFunction
context
- the context for drawing a snapshot of the operatorException
- Thrown, if state could not be created ot restored.@VisibleForTesting protected com.amazonaws.services.kinesis.producer.KinesisProducer getKinesisProducer(com.amazonaws.services.kinesis.producer.KinesisProducerConfiguration producerConfig)
KinesisProducer
. Exposed so that tests can inject mock producers easily.Copyright © 2014–2021 The Apache Software Foundation. All rights reserved.