T - Type of the elements handled by this sinkpublic class ElasticsearchSink<T> extends ElasticsearchSinkBase<T>
ActionRequests
against a cluster for each incoming element.
The sink internally uses a TransportClient to communicate with an Elasticsearch cluster.
The sink will fail if no cluster can be connected to using the provided transport addresses passed to the constructor.
The Map passed to the constructor is used to create the TransportClient. The config keys can be found
in the Elasticsearch documentation. An important setting is cluster.name,
which should be set to the name of the cluster that the sink should emit to.
Internally, the sink will use a BulkProcessor to send ActionRequests.
This will buffer elements before sending a request to the cluster. The behaviour of the
BulkProcessor can be configured using these config keys:
bulk.flush.max.actions: Maximum amount of elements to buffer
bulk.flush.max.size.mb: Maximum amount of data (in megabytes) to buffer
bulk.flush.interval.ms: Interval at which to flush data regardless of the other two
settings in milliseconds
You also have to provide an ElasticsearchSinkFunction. This is used to create multiple
ActionRequests for each incoming element. See the class level documentation of
ElasticsearchSinkFunction for an example.
ElasticsearchSinkBase.BulkFlushBackoffPolicy, ElasticsearchSinkBase.FlushBackoffTypeSinkFunction.Context<T>CONFIG_KEY_BULK_FLUSH_BACKOFF_DELAY, CONFIG_KEY_BULK_FLUSH_BACKOFF_ENABLE, CONFIG_KEY_BULK_FLUSH_BACKOFF_RETRIES, CONFIG_KEY_BULK_FLUSH_BACKOFF_TYPE, CONFIG_KEY_BULK_FLUSH_INTERVAL_MS, CONFIG_KEY_BULK_FLUSH_MAX_ACTIONS, CONFIG_KEY_BULK_FLUSH_MAX_SIZE_MB| Constructor and Description |
|---|
ElasticsearchSink(Map<String,String> userConfig,
List<InetSocketAddress> transportAddresses,
ElasticsearchSinkFunction<T> elasticsearchSinkFunction)
Creates a new
ElasticsearchSink that connects to the cluster using a TransportClient. |
ElasticsearchSink(Map<String,String> userConfig,
List<InetSocketAddress> transportAddresses,
ElasticsearchSinkFunction<T> elasticsearchSinkFunction,
ActionRequestFailureHandler failureHandler)
Creates a new
ElasticsearchSink that connects to the cluster using a TransportClient. |
buildBulkProcessor, close, disableFlushOnCheckpoint, initializeState, invoke, open, snapshotStategetIterationRuntimeContext, getRuntimeContext, setRuntimeContextclone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitinvokepublic ElasticsearchSink(Map<String,String> userConfig, List<InetSocketAddress> transportAddresses, ElasticsearchSinkFunction<T> elasticsearchSinkFunction)
ElasticsearchSink that connects to the cluster using a TransportClient.userConfig - The map of user settings that are used when constructing the TransportClient and BulkProcessortransportAddresses - The addresses of Elasticsearch nodes to which to connect using a TransportClientelasticsearchSinkFunction - This is used to generate multiple ActionRequest from the incoming elementpublic ElasticsearchSink(Map<String,String> userConfig, List<InetSocketAddress> transportAddresses, ElasticsearchSinkFunction<T> elasticsearchSinkFunction, ActionRequestFailureHandler failureHandler)
ElasticsearchSink that connects to the cluster using a TransportClient.userConfig - The map of user settings that are used when constructing the TransportClient and BulkProcessortransportAddresses - The addresses of Elasticsearch nodes to which to connect using a TransportClientelasticsearchSinkFunction - This is used to generate multiple ActionRequest from the incoming elementfailureHandler - This is used to handle failed ActionRequestCopyright © 2014–2018 The Apache Software Foundation. All rights reserved.