ReplicatingInputFormat (flink 1.8-SNAPSHOT API)

java.lang.Object
- org.apache.flink.api.common.io.RichInputFormat<OT,S>
- - org.apache.flink.api.common.io.ReplicatingInputFormat<OT,S>

Type Parameters:

OT - The output type of the wrapped InputFormat.

S - The InputSplit type of the wrapped InputFormat.

All Implemented Interfaces:

Serializable, InputFormat<OT,S>, InputSplitSource<S>
```
@PublicEvolving
public final class ReplicatingInputFormat<OT,S extends InputSplit>
extends RichInputFormat<OT,S>
```
A ReplicatingInputFormat replicates any InputFormat to all parallel instances of a DataSource, i.e., the full input of the replicated InputFormat is completely processed by each parallel instance of the DataSource. This is done by assigning all InputSplits generated by the replicated InputFormat to each parallel instance. Replicated data can only be used as input for a InnerJoinOperatorBase or CrossOperatorBase with the same parallelism as the DataSource. Before being used as an input to a Join or Cross operator, replicated data might be processed in local pipelines by Map-based operators with the same parallelism as the source. Map-based operators are MapOperatorBase, FlatMapOperatorBase, FilterOperatorBase, and MapPartitionOperatorBase. Replicated DataSources can be used for local join processing (no data shipping) if one input is accessible on all parallel instance of a join and the other input is (randomly) partitioned across all parallel instances. However, a replicated DataSource is a plan hint that can invalidate a Flink program if not used correctly (see usage instructions above). In such situations, the optimizer is not able to generate a valid execution plan and the program execution will fail.

See Also:

InputFormat, RichInputFormat, InnerJoinOperatorBase, CrossOperatorBase, MapOperatorBase, FlatMapOperatorBase, FilterOperatorBase, MapPartitionOperatorBase, Serialized Form

Constructor Summary

Constructors
Constructor and Description

ReplicatingInputFormat(InputFormat<OT,S> wrappedIF)

Constructors
Constructor and Description
`ReplicatingInputFormat(InputFormat<OT,S> wrappedIF)`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`close()` Method that marks the end of the life-cycle of an input split.
`void`	`closeInputFormat()` Closes this InputFormat instance.
`void`	`configure(Configuration parameters)` Configures this input format.
`S[]`	`createInputSplits(int minNumSplits)` Creates the different splits of the input that can be processed in parallel.
`InputSplitAssigner`	`getInputSplitAssigner(S[] inputSplits)` Gets the type of the input splits that are processed by this input format.
`InputFormat<OT,S>`	`getReplicatedInputFormat()`
`RuntimeContext`	`getRuntimeContext()`
`BaseStatistics`	`getStatistics(BaseStatistics cachedStatistics)` Gets the basic statistics from the input described by this format.
`OT`	`nextRecord(OT reuse)` Reads the next record from the input.
`void`	`open(S split)` Opens a parallel instance of the input format to work on a split.
`void`	`openInputFormat()` Opens this InputFormat instance.
`boolean`	`reachedEnd()` Method used to check if the end of the input is reached.
`void`	`setRuntimeContext(RuntimeContext context)`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - ReplicatingInputFormat
```
public ReplicatingInputFormat(InputFormat<OT,S> wrappedIF)
```
- Method Detail
  - getReplicatedInputFormat
```
public InputFormat<OT,S> getReplicatedInputFormat()
```
  - configure
```
public void configure(Configuration parameters)
```
    Description copied from interface: InputFormat
    
    Configures this input format. Since input formats are instantiated generically and hence parameterless, this method is the place where the input formats set their basic fields based on configuration values.
    This method is always called first on a newly instantiated input format.
    
    Parameters:
    
    parameters - The configuration with all parameters (note: not the Flink config but the TaskConfig).
  - getStatistics
```
public BaseStatistics getStatistics(BaseStatistics cachedStatistics)
                             throws IOException
```
    Description copied from interface: InputFormat
    
    Gets the basic statistics from the input described by this format. If the input format does not know how to create those statistics, it may return null. This method optionally gets a cached version of the statistics. The input format may examine them and decide whether it directly returns them without spending effort to re-gather the statistics.
    When this method is called, the input format it guaranteed to be configured.
    
    Parameters:
    
    cachedStatistics - The statistics that were cached. May be null.
    
    Returns:
    
    The base statistics for the input, or null, if not available.
    
    Throws:
    
    IOException
  - createInputSplits
```
public S[] createInputSplits(int minNumSplits)
                      throws IOException
```
    Description copied from interface: InputFormat
    
    Creates the different splits of the input that can be processed in parallel.
    When this method is called, the input format it guaranteed to be configured.
    
    Parameters:
    
    minNumSplits - The minimum desired number of splits. If fewer are created, some parallel instances may remain idle.
    
    Returns:
    
    The splits of this input that can be processed in parallel.
    
    Throws:
    
    IOException - Thrown, when the creation of the splits was erroneous.
  - getInputSplitAssigner
```
public InputSplitAssigner getInputSplitAssigner(S[] inputSplits)
```
    Description copied from interface: InputFormat
    
    Gets the type of the input splits that are processed by this input format.
    
    Returns:
    
    The type of the input splits.
  - open
```
public void open(S split)
          throws IOException
```
    Description copied from interface: InputFormat
    
    Opens a parallel instance of the input format to work on a split.
    When this method is called, the input format it guaranteed to be configured.
    
    Parameters:
    
    split - The split to be opened.
    
    Throws:
    
    IOException - Thrown, if the spit could not be opened due to an I/O problem.
  - reachedEnd
```
public boolean reachedEnd()
                   throws IOException
```
    Description copied from interface: InputFormat
    
    Method used to check if the end of the input is reached.
    When this method is called, the input format it guaranteed to be opened.
    
    Returns:
    
    True if the end is reached, otherwise false.
    
    Throws:
    
    IOException - Thrown, if an I/O error occurred.
  - nextRecord
```
public OT nextRecord(OT reuse)
              throws IOException
```
    Description copied from interface: InputFormat
    
    Reads the next record from the input.
    When this method is called, the input format it guaranteed to be opened.
    
    Parameters:
    
    reuse - Object that may be reused.
    
    Returns:
    
    Read record.
    
    Throws:
    
    IOException - Thrown, if an I/O error occurred.
  - close
```
public void close()
           throws IOException
```
    Description copied from interface: InputFormat
    
    Method that marks the end of the life-cycle of an input split. Should be used to close channels and streams and release resources. After this method returns without an error, the input is assumed to be correctly read.
    When this method is called, the input format it guaranteed to be opened.
    
    Throws:
    
    IOException - Thrown, if the input could not be closed properly.
  - setRuntimeContext
```
public void setRuntimeContext(RuntimeContext context)
```
    Overrides:
    
    setRuntimeContext in class RichInputFormat<OT,S extends InputSplit>
  - getRuntimeContext
```
public RuntimeContext getRuntimeContext()
```
    Overrides:
    
    getRuntimeContext in class RichInputFormat<OT,S extends InputSplit>
  - openInputFormat
```
public void openInputFormat()
                     throws IOException
```
    Description copied from class: RichInputFormat
    
    Opens this InputFormat instance. This method is called once per parallel instance. Resources should be allocated in this method. (e.g. database connections, cache, etc.)
    
    Overrides:
    
    openInputFormat in class RichInputFormat<OT,S extends InputSplit>
    
    Throws:
    
    IOException - in case allocating the resources failed.
    
    See Also:
    
    InputFormat
  - closeInputFormat
```
public void closeInputFormat()
                      throws IOException
```
    Description copied from class: RichInputFormat
    
    Closes this InputFormat instance. This method is called once per parallel instance. Resources allocated during RichInputFormat.openInputFormat() should be closed in this method.
    
    Overrides:
    
    closeInputFormat in class RichInputFormat<OT,S extends InputSplit>
    
    Throws:
    
    IOException - in case closing the resources failed
    
    See Also:
    
    InputFormat

Back to Flink Website

Class ReplicatingInputFormat<OT,S extends InputSplit>

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Constructor Detail

ReplicatingInputFormat

Method Detail

getReplicatedInputFormat

configure

getStatistics

createInputSplits

getInputSplitAssigner

open

reachedEnd

nextRecord

close

setRuntimeContext

getRuntimeContext

openInputFormat

closeInputFormat

Back to Flink Website