org.apache.flink.streaming.connectors.elasticsearch

Class ElasticsearchSink<T>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.sink.RichSinkFunction<T>

org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSink<T>

Type Parameters:

T - Type of the elements emitted by this sink

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, SinkFunction<T>

public class ElasticsearchSink<T>
extends RichSinkFunction<T>

Sink that emits its input elements to an Elasticsearch cluster.

When using the first constructor ElasticsearchSink(java.util.Map, IndexRequestBuilder) the sink will create a local Node for communicating with the Elasticsearch cluster. When using the second constructor ElasticsearchSink(java.util.Map, java.util.List, IndexRequestBuilder) a TransportClient will be used instead.

Attention: When using the TransportClient the sink will fail if no cluster can be connected to. With the Node Client the sink will block and wait for a cluster to come online.

The Map passed to the constructor is forwarded to Elasticsearch when creating the Node or TransportClient. The config keys can be found in the Elasticsearch documentation. An important setting is cluster.name, this should be set to the name of the cluster that the sink should emit to.

Internally, the sink will use a BulkProcessor to send IndexRequests. This will buffer elements before sending a request to the cluster. The behaviour of the BulkProcessor can be configured using these config keys:

• bulk.flush.max.actions: Maximum amount of elements to buffer
• bulk.flush.max.size.mb: Maximum amount of data (in megabytes) to buffer
• bulk.flush.interval.ms: Interval at which to flush data regardless of the other two settings in milliseconds

You also have to provide an IndexRequestBuilder. This is used to create an IndexRequest from an element that needs to be added to Elasticsearch. See IndexRequestBuilder for an example.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static String	CONFIG_KEY_BULK_FLUSH_INTERVAL_MS
static String	CONFIG_KEY_BULK_FLUSH_MAX_ACTIONS
static String	CONFIG_KEY_BULK_FLUSH_MAX_SIZE_MB

Constructor Summary

Constructors

Constructor and Description
ElasticsearchSink(Map<String,String> userConfig, IndexRequestBuilder<T> indexRequestBuilder) Creates a new ElasticsearchSink that connects to the cluster using an embedded Node.
ElasticsearchSink(Map<String,String> userConfig, List<org.elasticsearch.common.transport.TransportAddress> transportNodes, IndexRequestBuilder<T> indexRequestBuilder) Creates a new ElasticsearchSink that connects to the cluster using a TransportClient.

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
void	invoke(T element)
void	open(org.apache.flink.configuration.Configuration configuration) Initializes the connection to Elasticsearch by either creating an embedded Node and retrieving the Client from it or by creating a TransportClient.

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CONFIG_KEY_BULK_FLUSH_MAX_ACTIONS

public static final String CONFIG_KEY_BULK_FLUSH_MAX_ACTIONS

See Also:

Constant Field Values

CONFIG_KEY_BULK_FLUSH_MAX_SIZE_MB

public static final String CONFIG_KEY_BULK_FLUSH_MAX_SIZE_MB

See Also:

Constant Field Values

CONFIG_KEY_BULK_FLUSH_INTERVAL_MS

public static final String CONFIG_KEY_BULK_FLUSH_INTERVAL_MS

See Also:

Constant Field Values

Constructor Detail

ElasticsearchSink

public ElasticsearchSink(Map<String,String> userConfig,
                 IndexRequestBuilder<T> indexRequestBuilder)

Creates a new ElasticsearchSink that connects to the cluster using an embedded Node.

Parameters:

userConfig - The map of user settings that are passed when constructing the Node and BulkProcessor

indexRequestBuilder - This is used to generate the IndexRequest from the incoming element

ElasticsearchSink

public ElasticsearchSink(Map<String,String> userConfig,
                 List<org.elasticsearch.common.transport.TransportAddress> transportNodes,
                 IndexRequestBuilder<T> indexRequestBuilder)

Creates a new ElasticsearchSink that connects to the cluster using a TransportClient.

Parameters:

userConfig - The map of user settings that are passed when constructing the TransportClient and BulkProcessor

transportNodes - The Elasticsearch Nodes to which to connect using a TransportClient

indexRequestBuilder - This is used to generate the IndexRequest from the incoming element

Method Detail

open

public void open(org.apache.flink.configuration.Configuration configuration)

Initializes the connection to Elasticsearch by either creating an embedded Node and retrieving the Client from it or by creating a TransportClient.

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.api.common.functions.AbstractRichFunction

invoke

public void invoke(T element)

Specified by:

invoke in interface SinkFunction<T>

Specified by:

invoke in class RichSinkFunction<T>

close

public void close()

Specified by:

close in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

close in class org.apache.flink.api.common.functions.AbstractRichFunction