org.apache.flink.hadoopcompatibility.mapred

Class HadoopReduceCombineFunction<KEYIN,VALUEIN,KEYOUT,VALUEOUT>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.api.common.functions.RichGroupReduceFunction<Tuple2<KEYIN,VALUEIN>,Tuple2<KEYOUT,VALUEOUT>>

org.apache.flink.hadoopcompatibility.mapred.HadoopReduceCombineFunction<KEYIN,VALUEIN,KEYOUT,VALUEOUT>

All Implemented Interfaces:

Serializable, Function, GroupCombineFunction<Tuple2<KEYIN,VALUEIN>,Tuple2<KEYIN,VALUEIN>>, GroupReduceFunction<Tuple2<KEYIN,VALUEIN>,Tuple2<KEYOUT,VALUEOUT>>, RichFunction, ResultTypeQueryable<Tuple2<KEYOUT,VALUEOUT>>

@Public
public final class HadoopReduceCombineFunction<KEYIN,VALUEIN,KEYOUT,VALUEOUT>
extends RichGroupReduceFunction<Tuple2<KEYIN,VALUEIN>,Tuple2<KEYOUT,VALUEOUT>>
implements GroupCombineFunction<Tuple2<KEYIN,VALUEIN>,Tuple2<KEYIN,VALUEIN>>, ResultTypeQueryable<Tuple2<KEYOUT,VALUEOUT>>, Serializable

This wrapper maps a Hadoop Reducer and Combiner (mapred API) to a combinable Flink GroupReduceFunction.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
HadoopReduceCombineFunction(org.apache.hadoop.mapred.Reducer<KEYIN,VALUEIN,KEYOUT,VALUEOUT> hadoopReducer, org.apache.hadoop.mapred.Reducer<KEYIN,VALUEIN,KEYIN,VALUEIN> hadoopCombiner) Maps two Hadoop Reducer (mapred API) to a combinable Flink GroupReduceFunction.
HadoopReduceCombineFunction(org.apache.hadoop.mapred.Reducer<KEYIN,VALUEIN,KEYOUT,VALUEOUT> hadoopReducer, org.apache.hadoop.mapred.Reducer<KEYIN,VALUEIN,KEYIN,VALUEIN> hadoopCombiner, org.apache.hadoop.mapred.JobConf conf) Maps two Hadoop Reducer (mapred API) to a combinable Flink GroupReduceFunction.

Method Summary

Modifier and Type	Method and Description
void	combine(Iterable<Tuple2<KEYIN,VALUEIN>> values, Collector<Tuple2<KEYIN,VALUEIN>> out) The combine method, called (potentially multiple timed) with subgroups of elements.
TypeInformation<Tuple2<KEYOUT,VALUEOUT>>	getProducedType() Gets the data type (as a TypeInformation) produced by this function or input format.
void	open(OpenContext openContext) Initialization method for the function.
void	reduce(Iterable<Tuple2<KEYIN,VALUEIN>> values, Collector<Tuple2<KEYOUT,VALUEOUT>> out) The reduce method.

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

close, getIterationRuntimeContext, getRuntimeContext, open, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopReduceCombineFunction

public HadoopReduceCombineFunction(org.apache.hadoop.mapred.Reducer<KEYIN,VALUEIN,KEYOUT,VALUEOUT> hadoopReducer,
                                   org.apache.hadoop.mapred.Reducer<KEYIN,VALUEIN,KEYIN,VALUEIN> hadoopCombiner)

Maps two Hadoop Reducer (mapred API) to a combinable Flink GroupReduceFunction.

Parameters:

hadoopReducer - The Hadoop Reducer that is mapped to a GroupReduceFunction.

hadoopCombiner - The Hadoop Reducer that is mapped to the combiner function.

HadoopReduceCombineFunction

public HadoopReduceCombineFunction(org.apache.hadoop.mapred.Reducer<KEYIN,VALUEIN,KEYOUT,VALUEOUT> hadoopReducer,
                                   org.apache.hadoop.mapred.Reducer<KEYIN,VALUEIN,KEYIN,VALUEIN> hadoopCombiner,
                                   org.apache.hadoop.mapred.JobConf conf)

Maps two Hadoop Reducer (mapred API) to a combinable Flink GroupReduceFunction.

Parameters:

hadoopReducer - The Hadoop Reducer that is mapped to a GroupReduceFunction.

hadoopCombiner - The Hadoop Reducer that is mapped to the combiner function.

conf - The JobConf that is used to configure both Hadoop Reducers.

Method Detail

open

@PublicEvolving
public void open(OpenContext openContext)
                          throws Exception

Description copied from interface: RichFunction

Initialization method for the function. It is called before the actual working methods (like map or join) and thus suitable for one time setup work. For functions that are part of an iteration, this method will be invoked at the beginning of each iteration superstep.

The openContext object passed to the function can be used for configuration and initialization. The openContext contains some necessary information that were configured on the function in the program composition.

 public class MyFilter extends RichFilterFunction<String> {

     private String searchString;

     public void open(OpenContext openContext) {
         // initialize the value of searchString
     }

     public boolean filter(String value) {
         return value.equals(searchString);
     }
 }
 

By default, this method does nothing.

1. If you implement open(OpenContext openContext), the open(OpenContext openContext) will be invoked and the open(Configuration parameters) won't be invoked. 2. If you don't implement open(OpenContext openContext), the open(Configuration parameters) will be invoked in the default implementation of the open(OpenContext openContext).

Specified by:

open in interface RichFunction

Parameters:

openContext - The context containing information about the context in which the function is opened.

Throws:

Exception - Implementations may forward exceptions, which are caught by the runtime. When the runtime catches an exception, it aborts the task and lets the fail-over logic decide whether to retry the task execution.

reduce

public void reduce(Iterable<Tuple2<KEYIN,VALUEIN>> values,
                   Collector<Tuple2<KEYOUT,VALUEOUT>> out)
            throws Exception

Description copied from interface: GroupReduceFunction

The reduce method. The function receives one call per group of elements.

Specified by:

reduce in interface GroupReduceFunction<Tuple2<KEYIN,VALUEIN>,Tuple2<KEYOUT,VALUEOUT>>

Specified by:

reduce in class RichGroupReduceFunction<Tuple2<KEYIN,VALUEIN>,Tuple2<KEYOUT,VALUEOUT>>

Parameters:

values - All records that belong to the given input key.

out - The collector to hand results to.

Throws:

Exception - This method may throw exceptions. Throwing an exception will cause the operation to fail and may trigger recovery.

combine

public void combine(Iterable<Tuple2<KEYIN,VALUEIN>> values,
                    Collector<Tuple2<KEYIN,VALUEIN>> out)
             throws Exception

Description copied from interface: GroupCombineFunction

The combine method, called (potentially multiple timed) with subgroups of elements.

Specified by:

combine in interface GroupCombineFunction<Tuple2<KEYIN,VALUEIN>,Tuple2<KEYIN,VALUEIN>>

Parameters:

values - The elements to be combined.

out - The collector to use to return values from the function.

Throws:

Exception - The function may throw Exceptions, which will cause the program to cancel, and may trigger the recovery logic.

getProducedType

public TypeInformation<Tuple2<KEYOUT,VALUEOUT>> getProducedType()

Description copied from interface: ResultTypeQueryable

Gets the data type (as a TypeInformation) produced by this function or input format.

Specified by:

getProducedType in interface ResultTypeQueryable<Tuple2<KEYOUT,VALUEOUT>>

Returns:

The data type produced by this function or input format.