org.apache.flink.batch.tests

Class Generator

java.lang.Object

org.apache.flink.batch.tests.Generator

All Implemented Interfaces:

Serializable, InputFormat<Tuple2<String,Integer>,GenericInputSplit>, InputSplitSource<GenericInputSplit>

public class Generator
extends Object
implements InputFormat<Tuple2<String,Integer>,GenericInputSplit>

InputFormat that generates a deterministic DataSet of Tuple2(String, Integer)

• String: key, can be repeated.
• Integer: uniformly distributed int between 0 and 127

See Also:

Serialized Form

Method Summary

Modifier and Type	Method and Description
void	close() Method that marks the end of the life-cycle of an input split.
void	configure(Configuration parameters) Configures this input format.
GenericInputSplit[]	createInputSplits(int minNumSplits) Computes the input splits.
static Generator	generate(long numKeys, int recordsPerKey)
static Generator	generateInfinitely(long numKeys)
InputSplitAssigner	getInputSplitAssigner(GenericInputSplit[] inputSplits) Returns the assigner for the input splits.
BaseStatistics	getStatistics(BaseStatistics cachedStatistics) Gets the basic statistics from the input described by this format.
Tuple2<String,Integer>	nextRecord(Tuple2<String,Integer> reuse) Reads the next record from the input.
void	open(GenericInputSplit split) Opens a parallel instance of the input format to work on a split.
boolean	reachedEnd() Method used to check if the end of the input is reached.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

generate

public static Generator generate(long numKeys,
                                 int recordsPerKey)

generateInfinitely

public static Generator generateInfinitely(long numKeys)

configure

public void configure(Configuration parameters)

Description copied from interface: InputFormat

Configures this input format. Since input formats are instantiated generically and hence parameterless, this method is the place where the input formats set their basic fields based on configuration values.

This method is always called first on a newly instantiated input format.

Specified by:

configure in interface InputFormat<Tuple2<String,Integer>,GenericInputSplit>

Parameters:

parameters - The configuration with all parameters (note: not the Flink config but the TaskConfig).

getStatistics

public BaseStatistics getStatistics(BaseStatistics cachedStatistics)

Description copied from interface: InputFormat

Gets the basic statistics from the input described by this format. If the input format does not know how to create those statistics, it may return null. This method optionally gets a cached version of the statistics. The input format may examine them and decide whether it directly returns them without spending effort to re-gather the statistics.

When this method is called, the input format is guaranteed to be configured.

Specified by:

getStatistics in interface InputFormat<Tuple2<String,Integer>,GenericInputSplit>

Parameters:

cachedStatistics - The statistics that were cached. May be null.

Returns:

The base statistics for the input, or null, if not available.

createInputSplits

public GenericInputSplit[] createInputSplits(int minNumSplits)

Description copied from interface: InputSplitSource

Computes the input splits. The given minimum number of splits is a hint as to how many splits are desired.

Specified by:

createInputSplits in interface InputFormat<Tuple2<String,Integer>,GenericInputSplit>

Specified by:

createInputSplits in interface InputSplitSource<GenericInputSplit>

Parameters:

minNumSplits - Number of minimal input splits, as a hint.

Returns:

An array of input splits.

getInputSplitAssigner

public InputSplitAssigner getInputSplitAssigner(GenericInputSplit[] inputSplits)

Description copied from interface: InputSplitSource

Returns the assigner for the input splits. Assigner determines which parallel instance of the input format gets which input split.

Specified by:

getInputSplitAssigner in interface InputFormat<Tuple2<String,Integer>,GenericInputSplit>

Specified by:

getInputSplitAssigner in interface InputSplitSource<GenericInputSplit>

Returns:

The input split assigner.

open

public void open(GenericInputSplit split)
          throws IOException

Description copied from interface: InputFormat

Opens a parallel instance of the input format to work on a split.

When this method is called, the input format it guaranteed to be configured.

Specified by:

open in interface InputFormat<Tuple2<String,Integer>,GenericInputSplit>

Parameters:

split - The split to be opened.

Throws:

IOException - Thrown, if the spit could not be opened due to an I/O problem.

reachedEnd

public boolean reachedEnd()

Description copied from interface: InputFormat

Method used to check if the end of the input is reached.

When this method is called, the input format it guaranteed to be opened.

Specified by:

reachedEnd in interface InputFormat<Tuple2<String,Integer>,GenericInputSplit>

Returns:

True if the end is reached, otherwise false.

nextRecord

public Tuple2<String,Integer> nextRecord(Tuple2<String,Integer> reuse)
                                  throws IOException

Description copied from interface: InputFormat

Reads the next record from the input.

When this method is called, the input format it guaranteed to be opened.

Specified by:

nextRecord in interface InputFormat<Tuple2<String,Integer>,GenericInputSplit>

Parameters:

reuse - Object that may be reused.

Returns:

Read record.

Throws:

IOException - Thrown, if an I/O error occurred.

close

public void close()

Description copied from interface: InputFormat

Method that marks the end of the life-cycle of an input split. Should be used to close channels and streams and release resources. After this method returns without an error, the input is assumed to be correctly read.

When this method is called, the input format it guaranteed to be opened.

Specified by:

close in interface InputFormat<Tuple2<String,Integer>,GenericInputSplit>