org.apache.flink.streaming.connectors.gcp.pubsub

Class PubSubSource<OUT>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.source.RichSourceFunction<OUT>

org.apache.flink.streaming.connectors.gcp.pubsub.PubSubSource<OUT>

All Implemented Interfaces:

Serializable, Function, RichFunction, ResultTypeQueryable<OUT>, CheckpointListener, ListCheckpointed<AcknowledgeIdsForCheckpoint<String>>, ParallelSourceFunction<OUT>, SourceFunction<OUT>

public class PubSubSource<OUT>
extends RichSourceFunction<OUT>
implements ResultTypeQueryable<OUT>, ParallelSourceFunction<OUT>, CheckpointListener, ListCheckpointed<AcknowledgeIdsForCheckpoint<String>>

PubSub Source, this Source will consume PubSub messages from a subscription and Acknowledge them on the next checkpoint. This ensures every message will get acknowledged at least once.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	PubSubSource.DeserializationSchemaBuilder Part of PubSubSource.PubSubSourceBuilder to set required fields.
static interface	PubSubSource.ProjectNameBuilder<OUT> Part of PubSubSource.PubSubSourceBuilder to set required fields.
static class	PubSubSource.PubSubSourceBuilder<OUT> Builder to create PubSubSource.
static interface	PubSubSource.SubscriptionNameBuilder<OUT> Part of PubSubSource.PubSubSourceBuilder to set required fields.

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

SourceFunction.SourceContext<T>

Field Summary

Fields

Modifier and Type	Field and Description
protected AcknowledgeOnCheckpoint<String>	acknowledgeOnCheckpoint
protected org.apache.flink.streaming.connectors.gcp.pubsub.PubSubSource.AcknowledgeOnCheckpointFactory	acknowledgeOnCheckpointFactory
protected com.google.auth.Credentials	credentials
protected PubSubDeserializationSchema<OUT>	deserializationSchema
protected boolean	isRunning
protected int	messagePerSecondRateLimit
protected PubSubSubscriberFactory	pubSubSubscriberFactory
protected FlinkConnectorRateLimiter	rateLimiter
protected PubSubSubscriber	subscriber

Method Summary

Modifier and Type	Method and Description
void	cancel() Cancels the source.
TypeInformation<OUT>	getProducedType() Gets the data type (as a TypeInformation) produced by this function or input format.
static PubSubSource.DeserializationSchemaBuilder	newBuilder()
void	notifyCheckpointAborted(long checkpointId) This method is called as a notification once a distributed checkpoint has been aborted.
void	notifyCheckpointComplete(long checkpointId) This method is called as a notification once a distributed checkpoint has been completed.
void	open(Configuration configuration) Initialization method for the function.
void	restoreState(List<AcknowledgeIdsForCheckpoint<String>> state) Restores the state of the function or operator to that of a previous checkpoint.
void	run(SourceFunction.SourceContext<OUT> sourceContext) Starts the source.
List<AcknowledgeIdsForCheckpoint<String>>	snapshotState(long checkpointId, long timestamp) Gets the current state of the function.

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

close, getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

deserializationSchema

protected final PubSubDeserializationSchema<OUT> deserializationSchema

pubSubSubscriberFactory

protected final PubSubSubscriberFactory pubSubSubscriberFactory

credentials

protected final com.google.auth.Credentials credentials

acknowledgeOnCheckpointFactory

protected final org.apache.flink.streaming.connectors.gcp.pubsub.PubSubSource.AcknowledgeOnCheckpointFactory acknowledgeOnCheckpointFactory

rateLimiter

protected final FlinkConnectorRateLimiter rateLimiter

messagePerSecondRateLimit

protected final int messagePerSecondRateLimit

acknowledgeOnCheckpoint

protected transient AcknowledgeOnCheckpoint<String> acknowledgeOnCheckpoint

subscriber

protected transient PubSubSubscriber subscriber

isRunning

protected transient volatile boolean isRunning

Method Detail

open

public void open(Configuration configuration)
          throws Exception

Description copied from interface: RichFunction

Initialization method for the function. It is called before the actual working methods (like map or join) and thus suitable for one time setup work. For functions that are part of an iteration, this method will be invoked at the beginning of each iteration superstep.

The configuration object passed to the function can be used for configuration and initialization. The configuration contains all parameters that were configured on the function in the program composition.

 public class MyFilter extends RichFilterFunction<String> {

     private String searchString;

     public void open(Configuration parameters) {
         this.searchString = parameters.getString("foo");
     }

     public boolean filter(String value) {
         return value.equals(searchString);
     }
 }
 

By default, this method does nothing.

Specified by:

open in interface RichFunction

Overrides:

open in class AbstractRichFunction

Parameters:

configuration - The configuration containing the parameters attached to the contract.

Throws:

Exception - Implementations may forward exceptions, which are caught by the runtime. When the runtime catches an exception, it aborts the task and lets the fail-over logic decide whether to retry the task execution.

See Also:

Configuration

run

public void run(SourceFunction.SourceContext<OUT> sourceContext)
         throws Exception

Description copied from interface: SourceFunction

Starts the source. Implementations can use the SourceFunction.SourceContext emit elements.

Sources that implement CheckpointedFunction must lock on the checkpoint lock (using a synchronized block) before updating internal state and emitting elements, to make both an atomic operation:

  public class ExampleCountSource implements SourceFunction<Long>, CheckpointedFunction {
      private long count = 0L;
      private volatile boolean isRunning = true;

      private transient ListState<Long> checkpointedCount;

      public void run(SourceContext<T> ctx) {
          while (isRunning && count < 1000) {
              // this synchronized block ensures that state checkpointing,
              // internal state updates and emission of elements are an atomic operation
              synchronized (ctx.getCheckpointLock()) {
                  ctx.collect(count);
                  count++;
              }
          }
      }

      public void cancel() {
          isRunning = false;
      }

      public void initializeState(FunctionInitializationContext context) {
          this.checkpointedCount = context
              .getOperatorStateStore()
              .getListState(new ListStateDescriptor<>("count", Long.class));

          if (context.isRestored()) {
              for (Long count : this.checkpointedCount.get()) {
                  this.count = count;
              }
          }
      }

      public void snapshotState(FunctionSnapshotContext context) {
          this.checkpointedCount.clear();
          this.checkpointedCount.add(count);
      }
 }
 

Specified by:

run in interface SourceFunction<OUT>

Parameters:

sourceContext - The context to emit elements to and for accessing locks.

Throws:

Exception

cancel

public void cancel()

Description copied from interface: SourceFunction

Cancels the source. Most sources will have a while loop inside the SourceFunction.run(SourceContext) method. The implementation needs to ensure that the source will break out of that loop after this method is called.

A typical pattern is to have an "volatile boolean isRunning" flag that is set to false in this method. That flag is checked in the loop condition.

When a source is canceled, the executing thread will also be interrupted (via Thread.interrupt()). The interruption happens strictly after this method has been called, so any interruption handler can rely on the fact that this method has completed. It is good practice to make any flags altered by this method "volatile", in order to guarantee the visibility of the effects of this method to any interruption handler.

Specified by:

cancel in interface SourceFunction<OUT>

getProducedType

public TypeInformation<OUT> getProducedType()

Description copied from interface: ResultTypeQueryable

Gets the data type (as a TypeInformation) produced by this function or input format.

Specified by:

getProducedType in interface ResultTypeQueryable<OUT>

Returns:

The data type produced by this function or input format.

newBuilder

public static PubSubSource.DeserializationSchemaBuilder newBuilder()

notifyCheckpointComplete

public void notifyCheckpointComplete(long checkpointId)
                              throws Exception

Description copied from interface: CheckpointListener

This method is called as a notification once a distributed checkpoint has been completed.

Note that any exception during this method will not cause the checkpoint to fail any more.

Specified by:

notifyCheckpointComplete in interface CheckpointListener

Parameters:

checkpointId - The ID of the checkpoint that has been completed.

Throws:

Exception - This method can propagate exceptions, which leads to a failure/recovery for the task. Not that this will NOT lead to the checkpoint being revoked.

notifyCheckpointAborted

public void notifyCheckpointAborted(long checkpointId)

Description copied from interface: CheckpointListener

This method is called as a notification once a distributed checkpoint has been aborted.

Specified by:

notifyCheckpointAborted in interface CheckpointListener

Parameters:

checkpointId - The ID of the checkpoint that has been aborted.

snapshotState

public List<AcknowledgeIdsForCheckpoint<String>> snapshotState(long checkpointId,
                                                               long timestamp)
                                                        throws Exception

Description copied from interface: ListCheckpointed

Gets the current state of the function. The state must reflect the result of all prior invocations to this function.

The returned list should contain one entry for redistributable unit of state. See the class docs for an illustration how list-style state redistribution works.

As special case, the returned list may be null or empty (if the operator has no state) or it may contain a single element (if the operator state is indivisible).

Specified by:

snapshotState in interface ListCheckpointed<AcknowledgeIdsForCheckpoint<String>>

Parameters:

checkpointId - The ID of the checkpoint - a unique and monotonously increasing value.

timestamp - The wall clock timestamp when the checkpoint was triggered by the master.

Returns:

The operator state in a list of redistributable, atomic sub-states. Should not return null, but empty list instead.

Throws:

Exception - Thrown if the creation of the state object failed. This causes the checkpoint to fail. The system may decide to fail the operation (and trigger recovery), or to discard this checkpoint attempt and to continue running and to try again with the next checkpoint attempt.

restoreState

public void restoreState(List<AcknowledgeIdsForCheckpoint<String>> state)
                  throws Exception

Description copied from interface: ListCheckpointed

Restores the state of the function or operator to that of a previous checkpoint. This method is invoked when the function is executed after a failure recovery. The state list may be empty if no state is to be recovered by the particular parallel instance of the function.

The given state list will contain all the sub states that this parallel instance of the function needs to handle. Refer to the class docs for an illustration how list-style state redistribution works.

Important: When implementing this interface together with RichFunction, then the restoreState() method is called before RichFunction.open(Configuration).

Specified by:

restoreState in interface ListCheckpointed<AcknowledgeIdsForCheckpoint<String>>

Parameters:

state - The state to be restored as a list of atomic sub-states.

Throws:

Exception - Throwing an exception in this method causes the recovery to fail. The exact consequence depends on the configured failure handling strategy, but typically the system will re-attempt the recovery, or try recovering from a different checkpoint.