IN
- Type of the messages to write into Kafka.public abstract class FlinkKafkaProducerBase<IN> extends RichSinkFunction<IN>
Modifier and Type | Field and Description |
---|---|
protected Exception |
asyncException
Errors encountered in the async producer are stored here
|
protected org.apache.kafka.clients.producer.Callback |
callback
The callback than handles error propagation or logging callbacks
|
static String |
KEY_DISABLE_METRICS
Configuration key for disabling the metrics reporting
|
protected boolean |
logFailuresOnly
Flag indicating whether to accept failures (and log them), or to fail on failures
|
protected KafkaPartitioner<IN> |
partitioner
User-provided partitioner for assigning an object to a Kafka partition.
|
protected int[] |
partitions
Array with the partition ids of the given topicId
The size of this array is the number of partitions
|
protected org.apache.kafka.clients.producer.KafkaProducer<byte[],byte[]> |
producer
KafkaProducer instance
|
protected Properties |
producerConfig
User defined properties for the Producer
|
protected KeyedSerializationSchema<IN> |
schema
(Serializable) SerializationSchema for turning objects used with Flink into
byte[] for Kafka.
|
protected String |
topicId
The name of the topic this producer is writing data to
|
Constructor and Description |
---|
FlinkKafkaProducerBase(String topicId,
KeyedSerializationSchema<IN> serializationSchema,
Properties producerConfig,
KafkaPartitioner<IN> customPartitioner)
The main constructor for creating a FlinkKafkaProducer.
|
Modifier and Type | Method and Description |
---|---|
protected void |
checkErroneous() |
void |
close()
Tear-down method for the user code.
|
static Properties |
getPropertiesFromBrokerList(String brokerList) |
void |
invoke(IN next)
Called when new data arrives to the sink, and forwards it to Kafka.
|
void |
open(Configuration configuration)
Initializes the connection to Kafka.
|
void |
setLogFailuresOnly(boolean logFailuresOnly)
Defines whether the producer should fail on errors, or only log them.
|
getIterationRuntimeContext, getRuntimeContext, setRuntimeContext
public static final String KEY_DISABLE_METRICS
protected final int[] partitions
protected final Properties producerConfig
protected final String topicId
protected final KeyedSerializationSchema<IN> schema
protected final KafkaPartitioner<IN> partitioner
protected boolean logFailuresOnly
protected transient org.apache.kafka.clients.producer.KafkaProducer<byte[],byte[]> producer
protected transient org.apache.kafka.clients.producer.Callback callback
protected transient volatile Exception asyncException
public FlinkKafkaProducerBase(String topicId, KeyedSerializationSchema<IN> serializationSchema, Properties producerConfig, KafkaPartitioner<IN> customPartitioner)
topicId
- The topic to write data toserializationSchema
- A serializable serialization schema for turning user objects into a kafka-consumable byte[] supporting key/value messagesproducerConfig
- Configuration properties for the KafkaProducer. 'bootstrap.servers.' is the only required argument.customPartitioner
- A serializable partitioner for assigning messages to Kafka partitions. Passing null will use Kafka's partitionerpublic void setLogFailuresOnly(boolean logFailuresOnly)
logFailuresOnly
- The flag to indicate logging-only on exceptions.public void open(Configuration configuration)
open
in interface RichFunction
open
in class AbstractRichFunction
configuration
- The configuration containing the parameters attached to the contract.Configuration
public void invoke(IN next) throws Exception
invoke
in interface SinkFunction<IN>
invoke
in class RichSinkFunction<IN>
next
- The incoming dataException
public void close() throws Exception
RichFunction
This method can be used for clean up work.
close
in interface RichFunction
close
in class AbstractRichFunction
Exception
- Implementations may forward exceptions, which are caught by the runtime. When the
runtime catches an exception, it aborts the task and lets the fail-over logic
decide whether to retry the task execution.public static Properties getPropertiesFromBrokerList(String brokerList)
Copyright © 2014–2017 The Apache Software Foundation. All rights reserved.