T
- Type of the elements handled by this sinkC
- Type of the Elasticsearch client, which implements AutoCloseable
@Internal public abstract class ElasticsearchSinkBase<T,C extends AutoCloseable> extends RichSinkFunction<T> implements CheckpointedFunction
This class implements the common behaviour across Elasticsearch versions, such as the use of
an internal BulkProcessor
to buffer multiple ActionRequest
s before sending the
requests to the cluster, as well as passing input records to the user provided ElasticsearchSinkFunction
for processing.
The version specific API calls for different Elasticsearch versions should be defined by a
concrete implementation of a ElasticsearchApiCallBridge
, which is provided to the
constructor of this class. This call bridge is used, for example, to create a Elasticsearch
Client
, handle failed item responses, etc.
Modifier and Type | Class and Description |
---|---|
static class |
ElasticsearchSinkBase.BulkFlushBackoffPolicy
Provides a backoff policy for bulk requests.
|
static class |
ElasticsearchSinkBase.FlushBackoffType
Used to control whether the retry delay should increase exponentially or remain constant.
|
SinkFunction.Context
Modifier and Type | Field and Description |
---|---|
static String |
CONFIG_KEY_BULK_FLUSH_BACKOFF_DELAY |
static String |
CONFIG_KEY_BULK_FLUSH_BACKOFF_ENABLE |
static String |
CONFIG_KEY_BULK_FLUSH_BACKOFF_RETRIES |
static String |
CONFIG_KEY_BULK_FLUSH_BACKOFF_TYPE |
static String |
CONFIG_KEY_BULK_FLUSH_INTERVAL_MS |
static String |
CONFIG_KEY_BULK_FLUSH_MAX_ACTIONS |
static String |
CONFIG_KEY_BULK_FLUSH_MAX_SIZE_MB |
Constructor and Description |
---|
ElasticsearchSinkBase(ElasticsearchApiCallBridge<C> callBridge,
Map<String,String> userConfig,
ElasticsearchSinkFunction<T> elasticsearchSinkFunction,
ActionRequestFailureHandler failureHandler) |
Modifier and Type | Method and Description |
---|---|
protected org.elasticsearch.action.bulk.BulkProcessor |
buildBulkProcessor(org.elasticsearch.action.bulk.BulkProcessor.Listener listener)
Build the
BulkProcessor . |
void |
close()
Tear-down method for the user code.
|
void |
disableFlushOnCheckpoint()
Disable flushing on checkpoint.
|
void |
initializeState(FunctionInitializationContext context)
This method is called when the parallel function instance is created during distributed
execution.
|
void |
invoke(T value,
SinkFunction.Context context)
Writes the given value to the sink.
|
void |
open(Configuration parameters)
Initialization method for the function.
|
void |
snapshotState(FunctionSnapshotContext context)
This method is called when a snapshot for a checkpoint is requested.
|
getIterationRuntimeContext, getRuntimeContext, setRuntimeContext
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
finish, invoke, writeWatermark
public static final String CONFIG_KEY_BULK_FLUSH_MAX_ACTIONS
public static final String CONFIG_KEY_BULK_FLUSH_MAX_SIZE_MB
public static final String CONFIG_KEY_BULK_FLUSH_INTERVAL_MS
public static final String CONFIG_KEY_BULK_FLUSH_BACKOFF_ENABLE
public static final String CONFIG_KEY_BULK_FLUSH_BACKOFF_TYPE
public static final String CONFIG_KEY_BULK_FLUSH_BACKOFF_RETRIES
public static final String CONFIG_KEY_BULK_FLUSH_BACKOFF_DELAY
public ElasticsearchSinkBase(ElasticsearchApiCallBridge<C> callBridge, Map<String,String> userConfig, ElasticsearchSinkFunction<T> elasticsearchSinkFunction, ActionRequestFailureHandler failureHandler)
public void disableFlushOnCheckpoint()
NOTE: If flushing on checkpoint is disabled, the Flink Elasticsearch Sink does NOT provide any strong guarantees for at-least-once delivery of action requests.
public void open(Configuration parameters) throws Exception
RichFunction
The configuration object passed to the function can be used for configuration and initialization. The configuration contains all parameters that were configured on the function in the program composition.
public class MyFilter extends RichFilterFunction<String> {
private String searchString;
public void open(Configuration parameters) {
this.searchString = parameters.getString("foo");
}
public boolean filter(String value) {
return value.equals(searchString);
}
}
By default, this method does nothing.
open
in interface RichFunction
open
in class AbstractRichFunction
parameters
- The configuration containing the parameters attached to the contract.Exception
- Implementations may forward exceptions, which are caught by the runtime.
When the runtime catches an exception, it aborts the task and lets the fail-over logic
decide whether to retry the task execution.Configuration
public void invoke(T value, SinkFunction.Context context) throws Exception
SinkFunction
You have to override this method when implementing a SinkFunction
, this is a
default
method for backward compatibility with the old-style method only.
invoke
in interface SinkFunction<T>
value
- The input record.context
- Additional context about the input record.Exception
- This method may throw exceptions. Throwing an exception will cause the
operation to fail and may trigger recovery.public void initializeState(FunctionInitializationContext context) throws Exception
CheckpointedFunction
initializeState
in interface CheckpointedFunction
context
- the context for initializing the operatorException
- Thrown, if state could not be created ot restored.public void snapshotState(FunctionSnapshotContext context) throws Exception
CheckpointedFunction
FunctionInitializationContext
when the Function was initialized, or offered now by FunctionSnapshotContext
itself.snapshotState
in interface CheckpointedFunction
context
- the context for drawing a snapshot of the operatorException
- Thrown, if state could not be created ot restored.public void close() throws Exception
RichFunction
This method can be used for clean up work.
close
in interface RichFunction
close
in class AbstractRichFunction
Exception
- Implementations may forward exceptions, which are caught by the runtime.
When the runtime catches an exception, it aborts the task and lets the fail-over logic
decide whether to retry the task execution.@VisibleForTesting protected org.elasticsearch.action.bulk.BulkProcessor buildBulkProcessor(org.elasticsearch.action.bulk.BulkProcessor.Listener listener)
BulkProcessor
.
Note: this is exposed for testing purposes.
Copyright © 2014–2022 The Apache Software Foundation. All rights reserved.