org.apache.flink.streaming.connectors.kafka

Class FlinkKafkaProducer<IN>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.sink.RichSinkFunction<IN>

org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction<IN,FlinkKafkaProducer.KafkaTransactionState,FlinkKafkaProducer.KafkaTransactionContext>

org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer<IN>

All Implemented Interfaces:

Serializable, Function, RichFunction, CheckpointListener, CheckpointedFunction, SinkFunction<IN>

Direct Known Subclasses:

FlinkKafkaShuffleProducer

Deprecated.

Please use KafkaSink.

@Deprecated
 @PublicEvolving
public class FlinkKafkaProducer<IN>
extends TwoPhaseCommitSinkFunction<IN,FlinkKafkaProducer.KafkaTransactionState,FlinkKafkaProducer.KafkaTransactionContext>

Flink Sink to produce data into a Kafka topic. By default producer will use FlinkKafkaProducer.Semantic.AT_LEAST_ONCE semantic. Before using FlinkKafkaProducer.Semantic.EXACTLY_ONCE please refer to Flink's Kafka connector documentation.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FlinkKafkaProducer.ContextStateSerializer Deprecated. TypeSerializer for FlinkKafkaProducer.KafkaTransactionContext.
static class	FlinkKafkaProducer.KafkaTransactionContext Deprecated. Context associated to this instance of the FlinkKafkaProducer.
static class	FlinkKafkaProducer.KafkaTransactionState Deprecated. State for handling transactions.
static class	FlinkKafkaProducer.NextTransactionalIdHint Deprecated. Keep information required to deduce next safe to use transactional id.
static class	FlinkKafkaProducer.NextTransactionalIdHintSerializer Deprecated. TypeSerializer for FlinkKafkaProducer.NextTransactionalIdHint.
static class	FlinkKafkaProducer.Semantic Deprecated. Semantics that can be chosen.
static class	FlinkKafkaProducer.TransactionStateSerializer Deprecated. TypeSerializer for FlinkKafkaProducer.KafkaTransactionState.

Nested classes/interfaces inherited from class org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction

TwoPhaseCommitSinkFunction.State<TXN,CONTEXT>, TwoPhaseCommitSinkFunction.StateSerializer<TXN,CONTEXT>, TwoPhaseCommitSinkFunction.StateSerializerSnapshot<TXN,CONTEXT>, TwoPhaseCommitSinkFunction.TransactionHolder<TXN>

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.sink.SinkFunction

SinkFunction.Context

Field Summary

Fields

Modifier and Type	Field and Description
protected Exception	asyncException Deprecated. Errors encountered in the async producer are stored here.
protected org.apache.kafka.clients.producer.Callback	callback Deprecated. The callback than handles error propagation or logging callbacks.
static int	DEFAULT_KAFKA_PRODUCERS_POOL_SIZE Deprecated. Default number of KafkaProducers in the pool.
static Time	DEFAULT_KAFKA_TRANSACTION_TIMEOUT Deprecated. Default value for kafka transaction timeout.
protected String	defaultTopicId Deprecated. The name of the default topic this producer is writing data to.
static String	KEY_DISABLE_METRICS Deprecated. Configuration key for disabling the metrics reporting.
protected AtomicLong	pendingRecords Deprecated. Number of unacknowledged records.
protected Properties	producerConfig Deprecated. User defined properties for the Producer.
static int	SAFE_SCALE_DOWN_FACTOR Deprecated. This coefficient determines what is the safe scale down factor.
protected FlinkKafkaProducer.Semantic	semantic Deprecated. Semantic chosen for this instance.
protected Map<String,int[]>	topicPartitionsMap Deprecated. Partitions of each topic.
protected boolean	writeTimestampToKafka Deprecated. Flag controlling whether we are writing the Flink record's timestamp into Kafka.

Fields inherited from class org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction

pendingCommitTransactions, state, userContext

Constructor Summary

Constructors

Constructor and Description
FlinkKafkaProducer(String defaultTopic, KafkaSerializationSchema<IN> serializationSchema, Properties producerConfig, FlinkKafkaProducer.Semantic semantic) Deprecated. Creates a FlinkKafkaProducer for a given topic.
FlinkKafkaProducer(String defaultTopic, KafkaSerializationSchema<IN> serializationSchema, Properties producerConfig, FlinkKafkaProducer.Semantic semantic, int kafkaProducersPoolSize) Deprecated. Creates a FlinkKafkaProducer for a given topic.
FlinkKafkaProducer(String topicId, KeyedSerializationSchema<IN> serializationSchema, Properties producerConfig) Deprecated. use FlinkKafkaProducer(String, KafkaSerializationSchema, Properties, FlinkKafkaProducer.Semantic)
FlinkKafkaProducer(String topicId, KeyedSerializationSchema<IN> serializationSchema, Properties producerConfig, FlinkKafkaProducer.Semantic semantic) Deprecated. use FlinkKafkaProducer(String, KafkaSerializationSchema, Properties, FlinkKafkaProducer.Semantic)
FlinkKafkaProducer(String defaultTopicId, KeyedSerializationSchema<IN> serializationSchema, Properties producerConfig, Optional<FlinkKafkaPartitioner<IN>> customPartitioner) Deprecated. use FlinkKafkaProducer(String, KafkaSerializationSchema, Properties, FlinkKafkaProducer.Semantic)
FlinkKafkaProducer(String defaultTopicId, KeyedSerializationSchema<IN> serializationSchema, Properties producerConfig, Optional<FlinkKafkaPartitioner<IN>> customPartitioner, FlinkKafkaProducer.Semantic semantic, int kafkaProducersPoolSize) Deprecated. use FlinkKafkaProducer(String, KafkaSerializationSchema, Properties, FlinkKafkaProducer.Semantic)
FlinkKafkaProducer(String topicId, SerializationSchema<IN> serializationSchema, Properties producerConfig) Deprecated. Creates a FlinkKafkaProducer for a given topic.
FlinkKafkaProducer(String topicId, SerializationSchema<IN> serializationSchema, Properties producerConfig, FlinkKafkaPartitioner<IN> customPartitioner, FlinkKafkaProducer.Semantic semantic, int kafkaProducersPoolSize) Deprecated. Creates a FlinkKafkaProducer for a given topic.
FlinkKafkaProducer(String topicId, SerializationSchema<IN> serializationSchema, Properties producerConfig, Optional<FlinkKafkaPartitioner<IN>> customPartitioner) Deprecated. Creates a FlinkKafkaProducer for a given topic.
FlinkKafkaProducer(String brokerList, String topicId, KeyedSerializationSchema<IN> serializationSchema) Deprecated. use FlinkKafkaProducer(String, KafkaSerializationSchema, Properties, FlinkKafkaProducer.Semantic)
FlinkKafkaProducer(String brokerList, String topicId, SerializationSchema<IN> serializationSchema) Deprecated. Creates a FlinkKafkaProducer for a given topic.

Method Summary

Modifier and Type	Method and Description
protected void	abort(FlinkKafkaProducer.KafkaTransactionState transaction) Deprecated. Abort a transaction.
protected void	acknowledgeMessage() Deprecated. ATTENTION to subclass implementors: When overriding this method, please always call super.acknowledgeMessage() to keep the invariants of the internal bookkeeping of the producer.
protected FlinkKafkaProducer.KafkaTransactionState	beginTransaction() Deprecated. Method that starts a new transaction.
protected void	checkErroneous() Deprecated.
void	close() Deprecated. Tear-down method for the user code.
protected void	commit(FlinkKafkaProducer.KafkaTransactionState transaction) Deprecated. Commit a pre-committed transaction.
protected FlinkKafkaInternalProducer<byte[],byte[]>	createProducer() Deprecated.
protected void	finishRecoveringContext(Collection<FlinkKafkaProducer.KafkaTransactionState> handledTransactions) Deprecated. Callback for subclasses which is called after restoring (each) user context.
protected static int[]	getPartitionsByTopic(String topic, org.apache.kafka.clients.producer.Producer<byte[],byte[]> producer) Deprecated.
static long	getTransactionTimeout(Properties producerConfig) Deprecated.
FlinkKafkaProducer<IN>	ignoreFailuresAfterTransactionTimeout() Deprecated. Disables the propagation of exceptions thrown when committing presumably timed out Kafka transactions during recovery of the job.
void	initializeState(FunctionInitializationContext context) Deprecated. This method is called when the parallel function instance is created during distributed execution.
protected Optional<FlinkKafkaProducer.KafkaTransactionContext>	initializeUserContext() Deprecated.
void	invoke(FlinkKafkaProducer.KafkaTransactionState transaction, IN next, SinkFunction.Context context) Deprecated. Write value within a transaction.
void	open(Configuration configuration) Deprecated. Initializes the connection to Kafka.
protected void	preCommit(FlinkKafkaProducer.KafkaTransactionState transaction) Deprecated. Pre commit previously created transaction.
protected void	recoverAndAbort(FlinkKafkaProducer.KafkaTransactionState transaction) Deprecated. Abort a transaction that was rejected by a coordinator after a failure.
protected void	recoverAndCommit(FlinkKafkaProducer.KafkaTransactionState transaction) Deprecated. Invoked on recovered transactions after a failure.
void	setLogFailuresOnly(boolean logFailuresOnly) Deprecated. Defines whether the producer should fail on errors, or only log them.
void	setTransactionalIdPrefix(String transactionalIdPrefix) Deprecated. Specifies the prefix of the transactional.id property to be used by the producers when communicating with Kafka.
void	setWriteTimestampToKafka(boolean writeTimestampToKafka) Deprecated. If set to true, Flink will write the (event time) timestamp attached to each record into Kafka.
void	snapshotState(FunctionSnapshotContext context) Deprecated. This method is called when a snapshot for a checkpoint is requested.

Methods inherited from class org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction

currentTransaction, enableTransactionTimeoutWarnings, finish, finishProcessing, getUserContext, invoke, invoke, notifyCheckpointAborted, notifyCheckpointComplete, pendingTransactions, setTransactionTimeout

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.functions.sink.SinkFunction

writeWatermark

Field Detail

SAFE_SCALE_DOWN_FACTOR

public static final int SAFE_SCALE_DOWN_FACTOR

Deprecated.

This coefficient determines what is the safe scale down factor.

If the Flink application previously failed before first checkpoint completed or we are starting new batch of FlinkKafkaProducer from scratch without clean shutdown of the previous one, FlinkKafkaProducer doesn't know what was the set of previously used Kafka's transactionalId's. In that case, it will try to play safe and abort all of the possible transactionalIds from the range of: [0, getNumberOfParallelSubtasks() * kafkaProducersPoolSize * SAFE_SCALE_DOWN_FACTOR)

The range of available to use transactional ids is: [0, getNumberOfParallelSubtasks() * kafkaProducersPoolSize)

This means that if we decrease getNumberOfParallelSubtasks() by a factor larger than SAFE_SCALE_DOWN_FACTOR we can have a left some lingering transaction.

See Also:

Constant Field Values

DEFAULT_KAFKA_PRODUCERS_POOL_SIZE

public static final int DEFAULT_KAFKA_PRODUCERS_POOL_SIZE

Deprecated.

Default number of KafkaProducers in the pool. See FlinkKafkaProducer.Semantic.EXACTLY_ONCE.

See Also:

Constant Field Values

DEFAULT_KAFKA_TRANSACTION_TIMEOUT

public static final Time DEFAULT_KAFKA_TRANSACTION_TIMEOUT

Deprecated.

Default value for kafka transaction timeout.

KEY_DISABLE_METRICS

public static final String KEY_DISABLE_METRICS

Deprecated.

Configuration key for disabling the metrics reporting.

See Also:

Constant Field Values

producerConfig

protected final Properties producerConfig

Deprecated.

User defined properties for the Producer.

defaultTopicId

protected final String defaultTopicId

Deprecated.

The name of the default topic this producer is writing data to.

topicPartitionsMap

protected final Map<String,int[]> topicPartitionsMap

Deprecated.

Partitions of each topic.

writeTimestampToKafka

protected boolean writeTimestampToKafka

Deprecated.

Flag controlling whether we are writing the Flink record's timestamp into Kafka.

semantic

protected FlinkKafkaProducer.Semantic semantic

Deprecated.

Semantic chosen for this instance.

callback

@Nullable
protected transient org.apache.kafka.clients.producer.Callback callback

Deprecated.

The callback than handles error propagation or logging callbacks.

asyncException

@Nullable
protected transient volatile Exception asyncException

Deprecated.

Errors encountered in the async producer are stored here.

pendingRecords

protected final AtomicLong pendingRecords

Deprecated.

Number of unacknowledged records.

Constructor Detail

FlinkKafkaProducer

public FlinkKafkaProducer(String brokerList,
                          String topicId,
                          SerializationSchema<IN> serializationSchema)

Deprecated.

Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to the topic.

Parameters:

brokerList - Comma separated addresses of the brokers

topicId - ID of the Kafka topic.

serializationSchema - User defined (keyless) serialization schema.

FlinkKafkaProducer

public FlinkKafkaProducer(String topicId,
                          SerializationSchema<IN> serializationSchema,
                          Properties producerConfig)

Deprecated.

Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to the topic.

Using this constructor, the default FlinkFixedPartitioner will be used as the partitioner. This default partitioner maps each sink subtask to a single Kafka partition (i.e. all records received by a sink subtask will end up in the same Kafka partition).

To use a custom partitioner, please use FlinkKafkaProducer(String, SerializationSchema, Properties, Optional) instead.

Parameters:

topicId - ID of the Kafka topic.

serializationSchema - User defined key-less serialization schema.

producerConfig - Properties with the producer configuration.

FlinkKafkaProducer

public FlinkKafkaProducer(String topicId,
                          SerializationSchema<IN> serializationSchema,
                          Properties producerConfig,
                          Optional<FlinkKafkaPartitioner<IN>> customPartitioner)

Deprecated.

Creates a FlinkKafkaProducer for a given topic. The sink produces its input to the topic. It accepts a key-less SerializationSchema and possibly a custom FlinkKafkaPartitioner.

Since a key-less SerializationSchema is used, all records sent to Kafka will not have an attached key. Therefore, if a partitioner is also not provided, records will be distributed to Kafka partitions in a round-robin fashion.

Parameters:

topicId - The topic to write data to

serializationSchema - A key-less serializable serialization schema for turning user objects into a kafka-consumable byte[]

producerConfig - Configuration properties for the KafkaProducer. 'bootstrap.servers.' is the only required argument.

customPartitioner - A serializable partitioner for assigning messages to Kafka partitions. If a partitioner is not provided, records will be distributed to Kafka partitions in a round-robin fashion.

FlinkKafkaProducer

public FlinkKafkaProducer(String topicId,
                          SerializationSchema<IN> serializationSchema,
                          Properties producerConfig,
                          @Nullable
                          FlinkKafkaPartitioner<IN> customPartitioner,
                          FlinkKafkaProducer.Semantic semantic,
                          int kafkaProducersPoolSize)

Deprecated.

Creates a FlinkKafkaProducer for a given topic. The sink produces its input to the topic. It accepts a key-less SerializationSchema and possibly a custom FlinkKafkaPartitioner.

Since a key-less SerializationSchema is used, all records sent to Kafka will not have an attached key. Therefore, if a partitioner is also not provided, records will be distributed to Kafka partitions in a round-robin fashion.

Parameters:

topicId - The topic to write data to

serializationSchema - A key-less serializable serialization schema for turning user objects into a kafka-consumable byte[]

producerConfig - Configuration properties for the KafkaProducer. 'bootstrap.servers.' is the only required argument.

customPartitioner - A serializable partitioner for assigning messages to Kafka partitions. If a partitioner is not provided, records will be distributed to Kafka partitions in a round-robin fashion.

semantic - Defines semantic that will be used by this producer (see FlinkKafkaProducer.Semantic).

kafkaProducersPoolSize - Overwrite default KafkaProducers pool size (see FlinkKafkaProducer.Semantic.EXACTLY_ONCE).

FlinkKafkaProducer

@Deprecated
public FlinkKafkaProducer(String brokerList,
                                      String topicId,
                                      KeyedSerializationSchema<IN> serializationSchema)

Deprecated. use FlinkKafkaProducer(String, KafkaSerializationSchema, Properties, FlinkKafkaProducer.Semantic)

Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to the topic.

Using this constructor, the default FlinkFixedPartitioner will be used as the partitioner. This default partitioner maps each sink subtask to a single Kafka partition (i.e. all records received by a sink subtask will end up in the same Kafka partition).

To use a custom partitioner, please use FlinkKafkaProducer(String, KeyedSerializationSchema, Properties, Optional) instead.

Parameters:

brokerList - Comma separated addresses of the brokers

topicId - ID of the Kafka topic.

serializationSchema - User defined serialization schema supporting key/value messages

FlinkKafkaProducer

@Deprecated
public FlinkKafkaProducer(String topicId,
                                      KeyedSerializationSchema<IN> serializationSchema,
                                      Properties producerConfig)

Deprecated. use FlinkKafkaProducer(String, KafkaSerializationSchema, Properties, FlinkKafkaProducer.Semantic)

Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to the topic.

Using this constructor, the default FlinkFixedPartitioner will be used as the partitioner. This default partitioner maps each sink subtask to a single Kafka partition (i.e. all records received by a sink subtask will end up in the same Kafka partition).

To use a custom partitioner, please use FlinkKafkaProducer(String, KeyedSerializationSchema, Properties, Optional) instead.

Parameters:

topicId - ID of the Kafka topic.

serializationSchema - User defined serialization schema supporting key/value messages

producerConfig - Properties with the producer configuration.

FlinkKafkaProducer

@Deprecated
public FlinkKafkaProducer(String topicId,
                                      KeyedSerializationSchema<IN> serializationSchema,
                                      Properties producerConfig,
                                      FlinkKafkaProducer.Semantic semantic)

Deprecated. use FlinkKafkaProducer(String, KafkaSerializationSchema, Properties, FlinkKafkaProducer.Semantic)

Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to the topic.

Using this constructor, the default FlinkFixedPartitioner will be used as the partitioner. This default partitioner maps each sink subtask to a single Kafka partition (i.e. all records received by a sink subtask will end up in the same Kafka partition).

Parameters:

topicId - ID of the Kafka topic.

serializationSchema - User defined serialization schema supporting key/value messages

producerConfig - Properties with the producer configuration.

semantic - Defines semantic that will be used by this producer (see FlinkKafkaProducer.Semantic).

FlinkKafkaProducer

@Deprecated
public FlinkKafkaProducer(String defaultTopicId,
                                      KeyedSerializationSchema<IN> serializationSchema,
                                      Properties producerConfig,
                                      Optional<FlinkKafkaPartitioner<IN>> customPartitioner)

Deprecated. use FlinkKafkaProducer(String, KafkaSerializationSchema, Properties, FlinkKafkaProducer.Semantic)

Creates a FlinkKafkaProducer for a given topic. The sink produces its input to the topic. It accepts a keyed KeyedSerializationSchema and possibly a custom FlinkKafkaPartitioner.

If a partitioner is not provided, written records will be partitioned by the attached key of each record (as determined by KeyedSerializationSchema.serializeKey(Object)). If written records do not have a key (i.e., KeyedSerializationSchema.serializeKey(Object) returns null), they will be distributed to Kafka partitions in a round-robin fashion.

Parameters:

defaultTopicId - The default topic to write data to

serializationSchema - A serializable serialization schema for turning user objects into a kafka-consumable byte[] supporting key/value messages

producerConfig - Configuration properties for the KafkaProducer. 'bootstrap.servers.' is the only required argument.

customPartitioner - A serializable partitioner for assigning messages to Kafka partitions. If a partitioner is not provided, records will be partitioned by the key of each record (determined by KeyedSerializationSchema.serializeKey(Object)). If the keys are null, then records will be distributed to Kafka partitions in a round-robin fashion.

FlinkKafkaProducer

@Deprecated
public FlinkKafkaProducer(String defaultTopicId,
                                      KeyedSerializationSchema<IN> serializationSchema,
                                      Properties producerConfig,
                                      Optional<FlinkKafkaPartitioner<IN>> customPartitioner,
                                      FlinkKafkaProducer.Semantic semantic,
                                      int kafkaProducersPoolSize)

Deprecated. use FlinkKafkaProducer(String, KafkaSerializationSchema, Properties, FlinkKafkaProducer.Semantic)

Creates a FlinkKafkaProducer for a given topic. The sink produces its input to the topic. It accepts a keyed KeyedSerializationSchema and possibly a custom FlinkKafkaPartitioner.

If a partitioner is not provided, written records will be partitioned by the attached key of each record (as determined by KeyedSerializationSchema.serializeKey(Object)). If written records do not have a key (i.e., KeyedSerializationSchema.serializeKey(Object) returns null), they will be distributed to Kafka partitions in a round-robin fashion.

Parameters:

defaultTopicId - The default topic to write data to

serializationSchema - A serializable serialization schema for turning user objects into a kafka-consumable byte[] supporting key/value messages

producerConfig - Configuration properties for the KafkaProducer. 'bootstrap.servers.' is the only required argument.

customPartitioner - A serializable partitioner for assigning messages to Kafka partitions. If a partitioner is not provided, records will be partitioned by the key of each record (determined by KeyedSerializationSchema.serializeKey(Object)). If the keys are null, then records will be distributed to Kafka partitions in a round-robin fashion.

semantic - Defines semantic that will be used by this producer (see FlinkKafkaProducer.Semantic).

kafkaProducersPoolSize - Overwrite default KafkaProducers pool size (see FlinkKafkaProducer.Semantic.EXACTLY_ONCE).

FlinkKafkaProducer

public FlinkKafkaProducer(String defaultTopic,
                          KafkaSerializationSchema<IN> serializationSchema,
                          Properties producerConfig,
                          FlinkKafkaProducer.Semantic semantic)

Deprecated.

Creates a FlinkKafkaProducer for a given topic. The sink produces its input to the topic. It accepts a KafkaSerializationSchema for serializing records to a ProducerRecord, including partitioning information.

Parameters:

defaultTopic - The default topic to write data to

serializationSchema - A serializable serialization schema for turning user objects into a kafka-consumable byte[] supporting key/value messages

producerConfig - Configuration properties for the KafkaProducer. 'bootstrap.servers.' is the only required argument.

semantic - Defines semantic that will be used by this producer (see FlinkKafkaProducer.Semantic).

FlinkKafkaProducer

public FlinkKafkaProducer(String defaultTopic,
                          KafkaSerializationSchema<IN> serializationSchema,
                          Properties producerConfig,
                          FlinkKafkaProducer.Semantic semantic,
                          int kafkaProducersPoolSize)

Deprecated.

Creates a FlinkKafkaProducer for a given topic. The sink produces its input to the topic. It accepts a KafkaSerializationSchema and possibly a custom FlinkKafkaPartitioner.

Parameters:

defaultTopic - The default topic to write data to

serializationSchema - A serializable serialization schema for turning user objects into a kafka-consumable byte[] supporting key/value messages

producerConfig - Configuration properties for the KafkaProducer. 'bootstrap.servers.' is the only required argument.

semantic - Defines semantic that will be used by this producer (see FlinkKafkaProducer.Semantic).

kafkaProducersPoolSize - Overwrite default KafkaProducers pool size (see FlinkKafkaProducer.Semantic.EXACTLY_ONCE).

Method Detail

setWriteTimestampToKafka

public void setWriteTimestampToKafka(boolean writeTimestampToKafka)

Deprecated.

If set to true, Flink will write the (event time) timestamp attached to each record into Kafka. Timestamps must be positive for Kafka to accept them.

Parameters:

writeTimestampToKafka - Flag indicating if Flink's internal timestamps are written to Kafka.

setLogFailuresOnly

public void setLogFailuresOnly(boolean logFailuresOnly)

Deprecated.

Defines whether the producer should fail on errors, or only log them. If this is set to true, then exceptions will be only logged, if set to false, exceptions will be eventually thrown and cause the streaming program to fail (and enter recovery).

Parameters:

logFailuresOnly - The flag to indicate logging-only on exceptions.

setTransactionalIdPrefix

public void setTransactionalIdPrefix(String transactionalIdPrefix)

Deprecated.

Specifies the prefix of the transactional.id property to be used by the producers when communicating with Kafka. If not set, the transactional.id will be prefixed with taskName + "-" + operatorUid.

Note that, if we change the prefix when the Flink application previously failed before first checkpoint completed or we are starting new batch of FlinkKafkaProducer from scratch without clean shutdown of the previous one, since we don't know what was the previously used transactional.id prefix, there will be some lingering transactions left.

Parameters:

transactionalIdPrefix - the transactional.id prefix

Throws:

NullPointerException - Thrown, if the transactionalIdPrefix was null.

ignoreFailuresAfterTransactionTimeout

public FlinkKafkaProducer<IN> ignoreFailuresAfterTransactionTimeout()

Deprecated.

Disables the propagation of exceptions thrown when committing presumably timed out Kafka transactions during recovery of the job. If a Kafka transaction is timed out, a commit will never be successful. Hence, use this feature to avoid recovery loops of the Job. Exceptions will still be logged to inform the user that data loss might have occurred.

Note that we use System.currentTimeMillis() to track the age of a transaction. Moreover, only exceptions thrown during the recovery are caught, i.e., the producer will attempt at least one commit of the transaction before giving up.

Overrides:

ignoreFailuresAfterTransactionTimeout in class TwoPhaseCommitSinkFunction<IN,FlinkKafkaProducer.KafkaTransactionState,FlinkKafkaProducer.KafkaTransactionContext>

open

public void open(Configuration configuration)
          throws Exception

Deprecated.

Initializes the connection to Kafka.

Specified by:

open in interface RichFunction

Overrides:

open in class AbstractRichFunction

Parameters:

configuration - The configuration containing the parameters attached to the contract.

Throws:

Exception - Implementations may forward exceptions, which are caught by the runtime. When the runtime catches an exception, it aborts the task and lets the fail-over logic decide whether to retry the task execution.

See Also:

Configuration

invoke

public void invoke(FlinkKafkaProducer.KafkaTransactionState transaction,
                   IN next,
                   SinkFunction.Context context)
            throws FlinkKafkaException

Deprecated.

Description copied from class: TwoPhaseCommitSinkFunction

Write value within a transaction.

Specified by:

invoke in class TwoPhaseCommitSinkFunction<IN,FlinkKafkaProducer.KafkaTransactionState,FlinkKafkaProducer.KafkaTransactionContext>

Throws:

FlinkKafkaException

close

public void close()
           throws FlinkKafkaException

Deprecated.

Description copied from interface: RichFunction

Tear-down method for the user code. It is called after the last call to the main working methods (e.g. map or join). For functions that are part of an iteration, this method will be invoked after each iteration superstep.

This method can be used for clean up work.

Specified by:

close in interface RichFunction

Overrides:

close in class TwoPhaseCommitSinkFunction<IN,FlinkKafkaProducer.KafkaTransactionState,FlinkKafkaProducer.KafkaTransactionContext>

Throws:

FlinkKafkaException

beginTransaction

protected FlinkKafkaProducer.KafkaTransactionState beginTransaction()
                                                             throws FlinkKafkaException

Deprecated.

Description copied from class: TwoPhaseCommitSinkFunction

Method that starts a new transaction.

Specified by:

beginTransaction in class TwoPhaseCommitSinkFunction<IN,FlinkKafkaProducer.KafkaTransactionState,FlinkKafkaProducer.KafkaTransactionContext>

Returns:

newly created transaction.

Throws:

FlinkKafkaException

preCommit

protected void preCommit(FlinkKafkaProducer.KafkaTransactionState transaction)
                  throws FlinkKafkaException

Deprecated.

Description copied from class: TwoPhaseCommitSinkFunction

Pre commit previously created transaction. Pre commit must make all of the necessary steps to prepare the transaction for a commit that might happen in the future. After this point the transaction might still be aborted, but underlying implementation must ensure that commit calls on already pre committed transactions will always succeed.

Usually implementation involves flushing the data.

Specified by:

preCommit in class TwoPhaseCommitSinkFunction<IN,FlinkKafkaProducer.KafkaTransactionState,FlinkKafkaProducer.KafkaTransactionContext>

Throws:

FlinkKafkaException

commit

protected void commit(FlinkKafkaProducer.KafkaTransactionState transaction)

Deprecated.

Description copied from class: TwoPhaseCommitSinkFunction

Commit a pre-committed transaction. If this method fail, Flink application will be restarted and TwoPhaseCommitSinkFunction.recoverAndCommit(Object) will be called again for the same transaction.

Specified by:

commit in class TwoPhaseCommitSinkFunction<IN,FlinkKafkaProducer.KafkaTransactionState,FlinkKafkaProducer.KafkaTransactionContext>

recoverAndCommit

protected void recoverAndCommit(FlinkKafkaProducer.KafkaTransactionState transaction)

Deprecated.

Description copied from class: TwoPhaseCommitSinkFunction

Invoked on recovered transactions after a failure. User implementation must ensure that this call will eventually succeed. If it fails, Flink application will be restarted and it will be invoked again. If it does not succeed eventually, a data loss will occur. Transactions will be recovered in an order in which they were created.

Overrides:

recoverAndCommit in class TwoPhaseCommitSinkFunction<IN,FlinkKafkaProducer.KafkaTransactionState,FlinkKafkaProducer.KafkaTransactionContext>

abort

protected void abort(FlinkKafkaProducer.KafkaTransactionState transaction)

Deprecated.

Description copied from class: TwoPhaseCommitSinkFunction

Abort a transaction.

Specified by:

abort in class TwoPhaseCommitSinkFunction<IN,FlinkKafkaProducer.KafkaTransactionState,FlinkKafkaProducer.KafkaTransactionContext>

recoverAndAbort

protected void recoverAndAbort(FlinkKafkaProducer.KafkaTransactionState transaction)

Deprecated.

Description copied from class: TwoPhaseCommitSinkFunction

Abort a transaction that was rejected by a coordinator after a failure.

Overrides:

recoverAndAbort in class TwoPhaseCommitSinkFunction<IN,FlinkKafkaProducer.KafkaTransactionState,FlinkKafkaProducer.KafkaTransactionContext>

acknowledgeMessage

protected void acknowledgeMessage()

Deprecated.

ATTENTION to subclass implementors: When overriding this method, please always call super.acknowledgeMessage() to keep the invariants of the internal bookkeeping of the producer. If not, be sure to know what you are doing.

snapshotState

public void snapshotState(FunctionSnapshotContext context)
                   throws Exception

Deprecated.

Description copied from interface: CheckpointedFunction

This method is called when a snapshot for a checkpoint is requested. This acts as a hook to the function to ensure that all state is exposed by means previously offered through FunctionInitializationContext when the Function was initialized, or offered now by FunctionSnapshotContext itself.

Specified by:

snapshotState in interface CheckpointedFunction

Overrides:

snapshotState in class TwoPhaseCommitSinkFunction<IN,FlinkKafkaProducer.KafkaTransactionState,FlinkKafkaProducer.KafkaTransactionContext>

Parameters:

context - the context for drawing a snapshot of the operator

Throws:

Exception - Thrown, if state could not be created ot restored.

initializeState

public void initializeState(FunctionInitializationContext context)
                     throws Exception

Deprecated.

Description copied from interface: CheckpointedFunction

This method is called when the parallel function instance is created during distributed execution. Functions typically set up their state storing data structures in this method.

Specified by:

initializeState in interface CheckpointedFunction

Overrides:

initializeState in class TwoPhaseCommitSinkFunction<IN,FlinkKafkaProducer.KafkaTransactionState,FlinkKafkaProducer.KafkaTransactionContext>

Parameters:

context - the context for initializing the operator

Throws:

Exception - Thrown, if state could not be created ot restored.

initializeUserContext

protected Optional<FlinkKafkaProducer.KafkaTransactionContext> initializeUserContext()

Deprecated.

Overrides:

initializeUserContext in class TwoPhaseCommitSinkFunction<IN,FlinkKafkaProducer.KafkaTransactionState,FlinkKafkaProducer.KafkaTransactionContext>

finishRecoveringContext

protected void finishRecoveringContext(Collection<FlinkKafkaProducer.KafkaTransactionState> handledTransactions)

Deprecated.

Description copied from class: TwoPhaseCommitSinkFunction

Callback for subclasses which is called after restoring (each) user context.

Overrides:

finishRecoveringContext in class TwoPhaseCommitSinkFunction<IN,FlinkKafkaProducer.KafkaTransactionState,FlinkKafkaProducer.KafkaTransactionContext>

Parameters:

handledTransactions - transactions which were already committed or aborted and do not need further handling

createProducer

protected FlinkKafkaInternalProducer<byte[],byte[]> createProducer()

Deprecated.

checkErroneous

protected void checkErroneous()
                       throws FlinkKafkaException

Deprecated.

Throws:

FlinkKafkaException

getPartitionsByTopic

protected static int[] getPartitionsByTopic(String topic,
                                            org.apache.kafka.clients.producer.Producer<byte[],byte[]> producer)

Deprecated.

getTransactionTimeout

public static long getTransactionTimeout(Properties producerConfig)

Deprecated.