T
- Type of the elements emitted by this sinkpublic class ElasticsearchSink<T> extends RichSinkFunction<T>
When using the first constructor ElasticsearchSink(java.util.Map, IndexRequestBuilder)
the sink will create a local Node
for communicating with the
Elasticsearch cluster. When using the second constructor
ElasticsearchSink(java.util.Map, java.util.List, IndexRequestBuilder)
a TransportClient
will
be used instead.
Attention: When using the TransportClient
the sink will fail if no cluster
can be connected to. With the Node Client
the sink will block and wait for a cluster
to come online.
The Map
passed to the constructor is forwarded to Elasticsearch when creating
the Node
or TransportClient
. The config keys can be found in the Elasticsearch
documentation. An important setting is cluster.name
, this should be set to the name
of the cluster that the sink should emit to.
Internally, the sink will use a BulkProcessor
to send IndexRequests
.
This will buffer elements before sending a request to the cluster. The behaviour of the
BulkProcessor
can be configured using these config keys:
bulk.flush.max.actions
: Maximum amount of elements to buffer
bulk.flush.max.size.mb
: Maximum amount of data (in megabytes) to buffer
bulk.flush.interval.ms
: Interval at which to flush data regardless of the other two
settings in milliseconds
You also have to provide an IndexRequestBuilder
. This is used to create an
IndexRequest
from an element that needs to be added to Elasticsearch. See
IndexRequestBuilder
for an example.
Modifier and Type | Field and Description |
---|---|
static String |
CONFIG_KEY_BULK_FLUSH_INTERVAL_MS |
static String |
CONFIG_KEY_BULK_FLUSH_MAX_ACTIONS |
static String |
CONFIG_KEY_BULK_FLUSH_MAX_SIZE_MB |
Constructor and Description |
---|
ElasticsearchSink(Map<String,String> userConfig,
IndexRequestBuilder<T> indexRequestBuilder)
Creates a new ElasticsearchSink that connects to the cluster using an embedded Node.
|
ElasticsearchSink(Map<String,String> userConfig,
List<org.elasticsearch.common.transport.TransportAddress> transportNodes,
IndexRequestBuilder<T> indexRequestBuilder)
Creates a new ElasticsearchSink that connects to the cluster using a TransportClient.
|
Modifier and Type | Method and Description |
---|---|
void |
close()
Tear-down method for the user code.
|
void |
invoke(T element)
Function for standard sink behaviour.
|
void |
open(Configuration configuration)
Initializes the connection to Elasticsearch by either creating an embedded
Node and retrieving the
Client from it or by creating a
TransportClient . |
getIterationRuntimeContext, getRuntimeContext, setRuntimeContext
public static final String CONFIG_KEY_BULK_FLUSH_MAX_ACTIONS
public static final String CONFIG_KEY_BULK_FLUSH_MAX_SIZE_MB
public static final String CONFIG_KEY_BULK_FLUSH_INTERVAL_MS
public ElasticsearchSink(Map<String,String> userConfig, IndexRequestBuilder<T> indexRequestBuilder)
userConfig
- The map of user settings that are passed when constructing the Node and BulkProcessorindexRequestBuilder
- This is used to generate the IndexRequest from the incoming elementpublic ElasticsearchSink(Map<String,String> userConfig, List<org.elasticsearch.common.transport.TransportAddress> transportNodes, IndexRequestBuilder<T> indexRequestBuilder)
userConfig
- The map of user settings that are passed when constructing the TransportClient and BulkProcessortransportNodes
- The Elasticsearch Nodes to which to connect using a TransportClient
indexRequestBuilder
- This is used to generate the IndexRequest from the incoming elementpublic void open(Configuration configuration)
Node
and retrieving the
Client
from it or by creating a
TransportClient
.open
in interface RichFunction
open
in class AbstractRichFunction
configuration
- The configuration containing the parameters attached to the contract.Configuration
public void invoke(T element)
SinkFunction
invoke
in interface SinkFunction<T>
invoke
in class RichSinkFunction<T>
element
- The input record.public void close()
RichFunction
This method can be used for clean up work.
close
in interface RichFunction
close
in class AbstractRichFunction
Copyright © 2014–2017 The Apache Software Foundation. All rights reserved.