org.apache.flink.api.scala

Class JoinDataSet<L,R>

java.lang.Object

org.apache.flink.api.scala.DataSet<scala.Tuple2<L,R>>

org.apache.flink.api.scala.JoinDataSet<L,R>

All Implemented Interfaces:

JoinFunctionAssigner<L,R>

public class JoinDataSet<L,R>
extends DataSet<scala.Tuple2<L,R>>
implements JoinFunctionAssigner<L,R>

A specific DataSet that results from a join operation. The result of a default join is a tuple containing the two values from the two sides of the join. The result of the join can be changed by specifying a custom join function using the apply method or by providing a RichFlatJoinFunction.

Example:

   val left = ...
   val right = ...
   val joinResult = left.join(right).where(0, 2).equalTo(0, 1) {
     (left, right) => new MyJoinResult(left, right)
   }
 

Or, using key selector functions with tuple data types:

   val left = ...
   val right = ...
   val joinResult = left.join(right).where({_._1}).equalTo({_._1) {
     (left, right) => new MyJoinResult(left, right)
   }
 

Constructor Summary

Constructors

Constructor and Description
JoinDataSet(JoinOperator.EquiJoin<L,R,scala.Tuple2<L,R>> defaultJoin, DataSet<L> leftInput, DataSet<R> rightInput, Keys<L> leftKeys, Keys<R> rightKeys)

Method Summary

Modifier and Type	Method and Description
<O> DataSet<O>	apply(FlatJoinFunction<L,R,O> joiner, TypeInformation<O> evidence$5, scala.reflect.ClassTag<O> evidence$6) Creates a new DataSet by passing each pair of joined values to the given function.
<O> DataSet<O>	apply(scala.Function2<L,R,O> fun, TypeInformation<O> evidence$1, scala.reflect.ClassTag<O> evidence$2) Creates a new DataSet where the result for each pair of joined elements is the result of the given function.
<O> DataSet<O>	apply(scala.Function3<L,R,Collector<O>,scala.runtime.BoxedUnit> fun, TypeInformation<O> evidence$3, scala.reflect.ClassTag<O> evidence$4) Creates a new DataSet by passing each pair of joined values to the given function.
<O> DataSet<O>	apply(JoinFunction<L,R,O> fun, TypeInformation<O> evidence$7, scala.reflect.ClassTag<O> evidence$8) Creates a new DataSet by passing each pair of joined values to the given function.
<K> Partitioner<K>	getPartitioner() Gets the custom partitioner used by this join, or null, if none is set.
<K> JoinDataSet<L,R>	withPartitioner(Partitioner<K> partitioner, TypeInformation<K> evidence$9)

Methods inherited from class org.apache.flink.api.scala.DataSet

aggregate, aggregate, clean, coGroup, collect, combineGroup, combineGroup, count, cross, crossWithHuge, crossWithTiny, distinct, distinct, distinct, distinct, filter, filter, first, flatMap, flatMap, flatMap, fullOuterJoin, fullOuterJoin, getExecutionEnvironment, getParallelism, getType, groupBy, groupBy, groupBy, iterate, iterateDelta, iterateDelta, iterateDelta, iterateDelta, iterateWithTermination, javaSet, join, join, joinWithHuge, joinWithTiny, leftOuterJoin, leftOuterJoin, map, map, mapPartition, mapPartition, mapPartition, max, max, min, min, name, output, partitionByHash, partitionByHash, partitionByHash, partitionByRange, partitionByRange, partitionByRange, partitionCustom, partitionCustom, partitionCustom, print, print, printOnTaskManager, printToErr, printToErr, rebalance, reduce, reduce, reduceGroup, reduceGroup, reduceGroup, registerAggregator, rightOuterJoin, rightOuterJoin, setParallelism, sortPartition, sortPartition, sortPartition, sum, sum, union, withBroadcastSet, withForwardedFields, withForwardedFieldsFirst, withForwardedFieldsSecond, withParameters, write, writeAsCsv, writeAsText

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

JoinDataSet

public JoinDataSet(JoinOperator.EquiJoin<L,R,scala.Tuple2<L,R>> defaultJoin,
                   DataSet<L> leftInput,
                   DataSet<R> rightInput,
                   Keys<L> leftKeys,
                   Keys<R> rightKeys)

Method Detail

apply

public <O> DataSet<O> apply(scala.Function2<L,R,O> fun,
                            TypeInformation<O> evidence$1,
                            scala.reflect.ClassTag<O> evidence$2)

Creates a new DataSet where the result for each pair of joined elements is the result of the given function.

Specified by:

apply in interface JoinFunctionAssigner<L,R>

apply

public <O> DataSet<O> apply(scala.Function3<L,R,Collector<O>,scala.runtime.BoxedUnit> fun,
                            TypeInformation<O> evidence$3,
                            scala.reflect.ClassTag<O> evidence$4)

Creates a new DataSet by passing each pair of joined values to the given function. The function can output zero or more elements using the Collector which will form the result.

Specified by:

apply in interface JoinFunctionAssigner<L,R>

apply

public <O> DataSet<O> apply(FlatJoinFunction<L,R,O> joiner,
                            TypeInformation<O> evidence$5,
                            scala.reflect.ClassTag<O> evidence$6)

Creates a new DataSet by passing each pair of joined values to the given function. The function can output zero or more elements using the Collector which will form the result.

A RichFlatJoinFunction can be used to access the broadcast variables and the RuntimeContext.

Specified by:

apply in interface JoinFunctionAssigner<L,R>

apply

public <O> DataSet<O> apply(JoinFunction<L,R,O> fun,
                            TypeInformation<O> evidence$7,
                            scala.reflect.ClassTag<O> evidence$8)

Creates a new DataSet by passing each pair of joined values to the given function. The function must output one value. The concatenation of those will be new the DataSet.

A RichJoinFunction can be used to access the broadcast variables and the RuntimeContext.

Specified by:

apply in interface JoinFunctionAssigner<L,R>

withPartitioner

public <K> JoinDataSet<L,R> withPartitioner(Partitioner<K> partitioner,
                                            TypeInformation<K> evidence$9)

Specified by:

withPartitioner in interface JoinFunctionAssigner<L,R>

getPartitioner

public <K> Partitioner<K> getPartitioner()

Gets the custom partitioner used by this join, or null, if none is set.