IN- The input type of the sink.
@PublicEvolving public class PulsarSink<IN> extends Object implements TwoPhaseCommittingSink<IN,PulsarCommittable>
PulsarSinkBuilderto construct a
PulsarSink. The following example shows how to create a PulsarSink receiving records of
PulsarSink<String> sink = PulsarSink.builder() .setServiceUrl(operator().serviceUrl()) .setAdminUrl(operator().adminUrl()) .setTopic(topic) .setSerializationSchema(PulsarSerializationSchema.pulsarSchema(Schema.STRING)) .build();
The sink supports all delivery guarantees described by
DeliveryGuarantee.NONEdoes not provide any guarantees: messages may be lost in case of issues on the Pulsar broker and messages may be duplicated in case of a Flink failure.
DeliveryGuarantee.AT_LEAST_ONCEthe sink will wait for all outstanding records in the Pulsar buffers to be acknowledged by the Pulsar producer on a checkpoint. No messages will be lost in case of any issue with the Pulsar brokers but messages may be duplicated when Flink restarts.
DeliveryGuarantee.EXACTLY_ONCE: In this mode the PulsarSink will write all messages in a Pulsar transaction that will be committed to Pulsar on a checkpoint. Thus, no duplicates will be seen in case of a Flink restart. However, this delays record writing effectively until a checkpoint is written, so adjust the checkpoint duration accordingly. Additionally, it is highly recommended to tweak Pulsar transaction timeout (link) >> maximum checkpoint duration + maximum restart duration or data loss may happen when Pulsar expires an uncommitted transaction.
PulsarSinkBuilder for more details.
|Modifier and Type||Method and Description|
Returns the serializer of the committable type.
public static <IN> PulsarSinkBuilder<IN> builder()
PulsarSinkBuilderto construct a new
IN- Type of incoming records.
@Internal public TwoPhaseCommittingSink.PrecommittingSinkWriter<IN,PulsarCommittable> createWriter(Sink.InitContext initContext)
TwoPhaseCommittingSink.PrecommittingSinkWriterthat creates committables on checkpoint or end of input.
initContext- the runtime context.
@Internal public Committer<PulsarCommittable> createCommitter()
Committerthat permanently makes the previously written data visible through
@Internal public SimpleVersionedSerializer<PulsarCommittable> getCommittableSerializer()
Copyright © 2014–2023 The Apache Software Foundation. All rights reserved.