package scala
The Flink Scala API. org.apache.flink.api.scala.ExecutionEnvironment is the starting-point of any Flink program. It can be used to read from local files, HDFS, or other sources. org.apache.flink.api.scala.DataSet is the main abstraction of data in Flink. It provides operations that create new DataSets via transformations. org.apache.flink.api.scala.GroupedDataSet provides operations on grouped data that results from org.apache.flink.api.scala.DataSet.groupBy().
Use org.apache.flink.api.scala.ExecutionEnvironment.getExecutionEnvironment to obtain an execution environment. This will either create a local environment or a remote environment, depending on the context where your program is executing.
- Deprecated
All Flink Scala APIs are deprecated and will be removed in a future Flink version. You can still build your application in Scala, but you should move to the Java version of either the DataStream and/or Table API.
- See also
- Alphabetic
- By Inheritance
- scala
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Type Members
-
class
SelectByMaxFunction[T] extends ReduceFunction[T]
SelectByMaxFunction to work with Scala tuples
SelectByMaxFunction to work with Scala tuples
- Annotations
- @Internal()
-
class
SelectByMinFunction[T] extends ReduceFunction[T]
SelectByMinFunction to work with Scala tuples
SelectByMinFunction to work with Scala tuples
- Annotations
- @Internal()
- class Tuple2CaseClassSerializer[T1, T2] extends ScalaCaseClassSerializer[(T1, T2)]
-
class
AggregateDataSet[T] extends DataSet[T]
The result of DataSet.aggregate.
The result of DataSet.aggregate. This can be used to chain more aggregations to the one aggregate operator.
- T
The type of the DataSet, i.e., the type of the elements of the DataSet.
- Annotations
- @deprecated @Public()
- Deprecated
(Since version 1.18.0)
- See also
-
class
CoGroupDataSet[L, R] extends DataSet[(Array[L], Array[R])]
A specific DataSet that results from a
coGroup
operation.A specific DataSet that results from a
coGroup
operation. The result of a default coGroup is a tuple containing two arrays of values from the two sides of the coGroup. The result of the coGroup can be changed by specifying a custom coGroup function using theapply
method or by providing a RichCoGroupFunction.Example:
val left = ... val right = ... val coGroupResult = left.coGroup(right).where(0, 2).isEqualTo(0, 1) { (left, right) => new MyCoGroupResult(left.min, right.max) }
Or, using key selector functions with tuple data types:
val left = ... val right = ... val coGroupResult = left.coGroup(right).where({_._1}).isEqualTo({_._1) { (left, right) => new MyCoGroupResult(left.max, right.min) }
- L
Type of the left input of the coGroup.
- R
Type of the right input of the coGroup.
- Annotations
- @deprecated @Public()
- Deprecated
(Since version 1.18.0)
- See also
-
class
CrossDataSet[L, R] extends DataSet[(L, R)]
A specific DataSet that results from a
cross
operation.A specific DataSet that results from a
cross
operation. The result of a default cross is a tuple containing the two values from the two sides of the cartesian product. The result of the cross can be changed by specifying a custom cross function using theapply
method or by providing a RichCrossFunction.Example:
val left = ... val right = ... val crossResult = left.cross(right) { (left, right) => new MyCrossResult(left, right) }
- L
Type of the left input of the cross.
- R
Type of the right input of the cross.
- Annotations
- @deprecated @Public()
- Deprecated
(Since version 1.18.0)
- See also
-
class
DataSet[T] extends AnyRef
The DataSet, the basic abstraction of Flink.
The DataSet, the basic abstraction of Flink. This represents a collection of elements of a specific type
T
. The operations in this class can be used to create new DataSets and to combine two DataSets. The methods of ExecutionEnvironment can be used to create a DataSet from an external source, such as files in HDFS. Thewrite*
methods can be used to write the elements to storage.All operations accept either a lambda function or an operation-specific function object for specifying the operation. For example, using a lambda:
val input: DataSet[String] = ... val mapped = input flatMap { _.split(" ") }
And using a
MapFunction
:val input: DataSet[String] = ... val mapped = input flatMap { new FlatMapFunction[String, String] { def flatMap(in: String, out: Collector[String]): Unit = { in.split(" ") foreach { out.collect(_) } } }
A rich function can be used when more control is required, for example for accessing the
RuntimeContext
. The rich function forflatMap
isRichFlatMapFunction
, all other functions are named similarly. All functions are available in packageorg.apache.flink.api.common.functions
.The elements are partitioned depending on the parallelism of the ExecutionEnvironment or of one specific DataSet.
Most of the operations have an implicit TypeInformation parameter. This is supplied by an implicit conversion in the
flink.api.scala
Package. For this to work, createTypeInformation needs to be imported. This is normally achieved with aimport org.apache.flink.api.scala._
- T
The type of the DataSet, i.e., the type of the elements of the DataSet.
- Annotations
- @deprecated @Public()
- Deprecated
(Since version 1.18.0)
- See also
-
class
ExecutionEnvironment extends AnyRef
The ExecutionEnvironment is the context in which a program is executed.
The ExecutionEnvironment is the context in which a program is executed. A local environment will cause execution in the current JVM, a remote environment will cause execution on a remote cluster installation.
The environment provides methods to control the job execution (such as setting the parallelism) and to interact with the outside world (data access).
To get an execution environment use the methods on the companion object:
- ExecutionEnvironment#getExecutionEnvironment
- ExecutionEnvironment#createLocalEnvironment
- ExecutionEnvironment#createRemoteEnvironment
Use ExecutionEnvironment#getExecutionEnvironment to get the correct environment depending on where the program is executed. If it is run inside an IDE a local environment will be created. If the program is submitted to a cluster a remote execution environment will be created.
- Annotations
- @deprecated @Public()
- Deprecated
(Since version 1.17.0)
- See also
-
class
GroupedDataSet[T] extends AnyRef
A DataSet to which a grouping key was added.
A DataSet to which a grouping key was added. Operations work on groups of elements with the same key (
aggregate
,reduce
, andreduceGroup
).A secondary sort order can be added with sortGroup, but this is only used when using one of the group-at-a-time operations, i.e.
reduceGroup
.- Annotations
- @deprecated @Public()
- Deprecated
(Since version 1.18.0)
- See also
-
class
JoinDataSet[L, R] extends DataSet[(L, R)] with JoinFunctionAssigner[L, R]
A specific DataSet that results from a
join
operation.A specific DataSet that results from a
join
operation. The result of a default join is a tuple containing the two values from the two sides of the join. The result of the join can be changed by specifying a custom join function using theapply
method or by providing a RichFlatJoinFunction.Example:
val left = ... val right = ... val joinResult = left.join(right).where(0, 2).equalTo(0, 1) { (left, right) => new MyJoinResult(left, right) }
Or, using key selector functions with tuple data types:
val left = ... val right = ... val joinResult = left.join(right).where({_._1}).equalTo({_._1) { (left, right) => new MyJoinResult(left, right) }
- L
Type of the left input of the join.
- R
Type of the right input of the join.
- Annotations
- @deprecated @Public()
- Deprecated
(Since version 1.18.0)
- See also
-
trait
JoinFunctionAssigner[L, R] extends AnyRef
- Annotations
- @deprecated @Public()
- Deprecated
(Since version 1.18.0)
-
class
PartitionSortedDataSet[T] extends DataSet[T]
The result of DataSet.sortPartition.
The result of DataSet.sortPartition. This can be used to append additional sort fields to the one sort-partition operator.
- T
The type of the DataSet, i.e., the type of the elements of the DataSet.
- Annotations
- @deprecated @Public()
- Deprecated
(Since version 1.18.0)
- See also
-
class
UnfinishedCoGroupOperation[L, R] extends UnfinishedKeyPairOperation[L, R, CoGroupDataSet[L, R]]
An unfinished coGroup operation that results from DataSet.coGroup The keys for the left and right side must be specified using first
where
and thenequalTo
.An unfinished coGroup operation that results from DataSet.coGroup The keys for the left and right side must be specified using first
where
and thenequalTo
. For example:val left = ... val right = ... val coGroupResult = left.coGroup(right).where(...).equalTo(...)
- L
The type of the left input of the coGroup.
- R
The type of the right input of the coGroup.
- Annotations
- @deprecated @Public()
- Deprecated
(Since version 1.18.0)
- See also
-
class
UnfinishedJoinOperation[L, R] extends UnfinishedJoinOperationBase[L, R, JoinDataSet[L, R]]
An unfinished inner join operation that results from calling DataSet.join().
An unfinished inner join operation that results from calling DataSet.join(). The keys for the left and right side must be specified using first
where
and thenequalTo
.For example:
val left = ... val right = ... val joinResult = left.join(right).where(...).equalTo(...)
- L
The type of the left input of the join.
- R
The type of the right input of the join.
- Annotations
- @deprecated @Public()
- Deprecated
(Since version 1.18.0)
-
class
UnfinishedOuterJoinOperation[L, R] extends UnfinishedJoinOperationBase[L, R, JoinFunctionAssigner[L, R]]
An unfinished outer join operation that results from calling, e.g.
An unfinished outer join operation that results from calling, e.g. DataSet.fullOuterJoin(). The keys for the left and right side must be specified using first
where
and thenequalTo
.Note that a join function must always be specified explicitly when construction an outer join operator.
For example:
val left = ... val right = ... val joinResult = left.fullOuterJoin(right).where(...).equalTo(...) { (first, second) => ... }
- L
The type of the left input of the join.
- R
The type of the right input of the join.
- Annotations
- @deprecated @Public()
- Deprecated
(Since version 1.18.0)
Value Members
- val FLIP_265_WARNING: String
- def createTuple2TypeInformation[T1, T2](t1: TypeInformation[T1], t2: TypeInformation[T2]): TypeInformation[(T1, T2)]
- implicit macro def createTypeInformation[T]: TypeInformation[T]
- def getCallLocationName(depth: Int = 3): String
- implicit val scalaNothingTypeInfo: TypeInformation[Nothing]
-
object
ClosureCleaner
- Annotations
- @Internal()
Deprecated Value Members
-
object
ExecutionEnvironment
- Annotations
- @deprecated @Public()
- Deprecated
(Since version 1.17.0)