org.apache.flink.api.scala

Class CoGroupDataSet<L,R>

java.lang.Object

org.apache.flink.api.scala.DataSet<scala.Tuple2<Object,Object>>

org.apache.flink.api.scala.CoGroupDataSet<L,R>

public class CoGroupDataSet<L,R>
extends DataSet<scala.Tuple2<Object,Object>>

A specific DataSet that results from a coGroup operation. The result of a default coGroup is a tuple containing two arrays of values from the two sides of the coGroup. The result of the coGroup can be changed by specifying a custom coGroup function using the apply method or by providing a RichCoGroupFunction.

Example:

   val left = ...
   val right = ...
   val coGroupResult = left.coGroup(right).where(0, 2).isEqualTo(0, 1) {
     (left, right) => new MyCoGroupResult(left.min, right.max)
   }
 

Or, using key selector functions with tuple data types:

   val left = ...
   val right = ...
   val coGroupResult = left.coGroup(right).where({_._1}).isEqualTo({_._1) {
     (left, right) => new MyCoGroupResult(left.max, right.min)
   }
 

Constructor Summary

Constructors

Constructor and Description
CoGroupDataSet(CoGroupOperator<L,R,scala.Tuple2<Object,Object>> defaultCoGroup, DataSet<L> leftInput, DataSet<R> rightInput, Keys<L> leftKeys, Keys<R> rightKeys)

Method Summary

Modifier and Type	Method and Description
<O> DataSet<O>	apply(CoGroupFunction<L,R,O> coGrouper, TypeInformation<O> evidence$5, scala.reflect.ClassTag<O> evidence$6) Creates a new DataSet by passing each pair of co-grouped element lists to the given function.
<O> DataSet<O>	apply(scala.Function2<scala.collection.Iterator<L>,scala.collection.Iterator<R>,O> fun, TypeInformation<O> evidence$1, scala.reflect.ClassTag<O> evidence$2) Creates a new DataSet where the result for each pair of co-grouped element lists is the result of the given function.
<O> DataSet<O>	apply(scala.Function3<scala.collection.Iterator<L>,scala.collection.Iterator<R>,Collector<O>,scala.runtime.BoxedUnit> fun, TypeInformation<O> evidence$3, scala.reflect.ClassTag<O> evidence$4) Creates a new DataSet where the result for each pair of co-grouped element lists is the result of the given function.
<K> Partitioner<K>	getPartitioner() Gets the custom partitioner used by this join, or null, if none is set.
CoGroupDataSet<L,R>	sortFirstGroup(int field, Order order) Adds a secondary sort key to the first input of this CoGroupDataSet.
CoGroupDataSet<L,R>	sortFirstGroup(String field, Order order) Adds a secondary sort key to the first input of this CoGroupDataSet.
CoGroupDataSet<L,R>	sortSecondGroup(int field, Order order) Adds a secondary sort key to the second input of this CoGroupDataSet.
CoGroupDataSet<L,R>	sortSecondGroup(String field, Order order) Adds a secondary sort key to the second input of this CoGroupDataSet.
<K> CoGroupDataSet<L,R>	withPartitioner(Partitioner<K> partitioner, TypeInformation<K> evidence$7)

Methods inherited from class org.apache.flink.api.scala.DataSet

aggregate, aggregate, clean, coGroup, collect, combineGroup, combineGroup, count, cross, crossWithHuge, crossWithTiny, distinct, distinct, distinct, distinct, filter, filter, first, flatMap, flatMap, flatMap, fullOuterJoin, fullOuterJoin, getExecutionEnvironment, getParallelism, getType, groupBy, groupBy, groupBy, iterate, iterateDelta, iterateDelta, iterateDelta, iterateDelta, iterateWithTermination, javaSet, join, join, joinWithHuge, joinWithTiny, leftOuterJoin, leftOuterJoin, map, map, mapPartition, mapPartition, mapPartition, max, max, min, min, name, output, partitionByHash, partitionByHash, partitionByHash, partitionByRange, partitionByRange, partitionByRange, partitionCustom, partitionCustom, partitionCustom, print, print, printOnTaskManager, printToErr, printToErr, rebalance, reduce, reduce, reduceGroup, reduceGroup, reduceGroup, registerAggregator, rightOuterJoin, rightOuterJoin, setParallelism, sortPartition, sortPartition, sortPartition, sum, sum, union, withBroadcastSet, withForwardedFields, withForwardedFieldsFirst, withForwardedFieldsSecond, withParameters, write, writeAsCsv, writeAsText

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CoGroupDataSet

public CoGroupDataSet(CoGroupOperator<L,R,scala.Tuple2<Object,Object>> defaultCoGroup,
                      DataSet<L> leftInput,
                      DataSet<R> rightInput,
                      Keys<L> leftKeys,
                      Keys<R> rightKeys)

Method Detail

apply

public <O> DataSet<O> apply(scala.Function2<scala.collection.Iterator<L>,scala.collection.Iterator<R>,O> fun,
                            TypeInformation<O> evidence$1,
                            scala.reflect.ClassTag<O> evidence$2)

Creates a new DataSet where the result for each pair of co-grouped element lists is the result of the given function.

apply

public <O> DataSet<O> apply(scala.Function3<scala.collection.Iterator<L>,scala.collection.Iterator<R>,Collector<O>,scala.runtime.BoxedUnit> fun,
                            TypeInformation<O> evidence$3,
                            scala.reflect.ClassTag<O> evidence$4)

Creates a new DataSet where the result for each pair of co-grouped element lists is the result of the given function. The function can output zero or more elements using the Collector which will form the result.

apply

public <O> DataSet<O> apply(CoGroupFunction<L,R,O> coGrouper,
                            TypeInformation<O> evidence$5,
                            scala.reflect.ClassTag<O> evidence$6)

Creates a new DataSet by passing each pair of co-grouped element lists to the given function. The function can output zero or more elements using the Collector which will form the result.

A RichCoGroupFunction can be used to access the broadcast variables and the RuntimeContext.

withPartitioner

public <K> CoGroupDataSet<L,R> withPartitioner(Partitioner<K> partitioner,
                                               TypeInformation<K> evidence$7)

getPartitioner

public <K> Partitioner<K> getPartitioner()

Gets the custom partitioner used by this join, or null, if none is set.

sortFirstGroup

public CoGroupDataSet<L,R> sortFirstGroup(int field,
                                          Order order)

Adds a secondary sort key to the first input of this CoGroupDataSet.

This only works on Tuple DataSets.

sortFirstGroup

public CoGroupDataSet<L,R> sortFirstGroup(String field,
                                          Order order)

Adds a secondary sort key to the first input of this CoGroupDataSet.

sortSecondGroup

public CoGroupDataSet<L,R> sortSecondGroup(int field,
                                           Order order)

Adds a secondary sort key to the second input of this CoGroupDataSet.

This only works on Tuple DataSets.

sortSecondGroup

public CoGroupDataSet<L,R> sortSecondGroup(String field,
                                           Order order)

Adds a secondary sort key to the second input of this CoGroupDataSet.