org.apache.flink.api.java.operators

Class DataSource<OUT>

java.lang.Object

org.apache.flink.api.java.DataSet<OUT>

org.apache.flink.api.java.operators.Operator<OUT,DataSource<OUT>>

org.apache.flink.api.java.operators.DataSource<OUT>

Type Parameters:

OUT - The type of the elements produced by this data source.

@Public
public class DataSource<OUT>
extends Operator<OUT,DataSource<OUT>>

An operation that creates a new data set (data source). The operation acts as the data set on which to apply further transformations. It encapsulates additional configuration parameters, to customize the execution.

Field Summary

Fields inherited from class org.apache.flink.api.java.operators.Operator

minResources, name, parallelism, preferredResources

Fields inherited from class org.apache.flink.api.java.DataSet

context

Constructor Summary

Constructors

Constructor and Description
DataSource(ExecutionEnvironment context, InputFormat<OUT,?> inputFormat, TypeInformation<OUT> type, String dataSourceLocationName) Creates a new data source.

Method Summary

Modifier and Type	Method and Description
InputFormat<OUT,?>	getInputFormat() Gets the input format that is executed by this data source.
Configuration	getParameters()
SplitDataProperties<OUT>	getSplitDataProperties() Returns the SplitDataProperties for the InputSplits of this DataSource for configurations.
protected GenericDataSourceBase<OUT,?>	translateToDataFlow()
DataSource<OUT>	withParameters(Configuration parameters) Pass a configuration to the InputFormat.

Methods inherited from class org.apache.flink.api.java.operators.Operator

getMinResources, getName, getParallelism, getPreferredResources, getResultType, name, setParallelism

Methods inherited from class org.apache.flink.api.java.DataSet

aggregate, checkSameExecutionContext, clean, coGroup, collect, combineGroup, count, cross, crossWithHuge, crossWithTiny, distinct, distinct, distinct, distinct, fillInType, filter, first, flatMap, fullOuterJoin, fullOuterJoin, getExecutionEnvironment, getType, groupBy, groupBy, groupBy, iterate, iterateDelta, join, join, joinWithHuge, joinWithTiny, leftOuterJoin, leftOuterJoin, map, mapPartition, max, maxBy, min, minBy, output, partitionByHash, partitionByHash, partitionByHash, partitionByRange, partitionByRange, partitionByRange, partitionCustom, partitionCustom, partitionCustom, print, print, printOnTaskManager, printToErr, printToErr, project, rebalance, reduce, reduceGroup, rightOuterJoin, rightOuterJoin, runOperation, sortPartition, sortPartition, sortPartition, sum, union, write, write, writeAsCsv, writeAsCsv, writeAsCsv, writeAsCsv, writeAsFormattedText, writeAsFormattedText, writeAsText, writeAsText

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DataSource

public DataSource(ExecutionEnvironment context,
                  InputFormat<OUT,?> inputFormat,
                  TypeInformation<OUT> type,
                  String dataSourceLocationName)

Creates a new data source.

Parameters:

context - The environment in which the data source gets executed.

inputFormat - The input format that the data source executes.

type - The type of the elements produced by this input format.

Method Detail

getInputFormat

@Internal
public InputFormat<OUT,?> getInputFormat()

Gets the input format that is executed by this data source.

Returns:

The input format that is executed by this data source.

withParameters

public DataSource<OUT> withParameters(Configuration parameters)

Pass a configuration to the InputFormat.

Parameters:

parameters - Configuration parameters

getParameters

public Configuration getParameters()

Returns:

Configuration for the InputFormat.

getSplitDataProperties

@PublicEvolving
public SplitDataProperties<OUT> getSplitDataProperties()

Returns the SplitDataProperties for the InputSplits of this DataSource for configurations.

SplitDataProperties can help to generate more efficient execution plans.

IMPORTANT: Incorrect configuration of SplitDataProperties can cause wrong results!

Returns:

The SplitDataProperties for the InputSplits of this DataSource.

translateToDataFlow

protected GenericDataSourceBase<OUT,?> translateToDataFlow()