org.apache.flink.hadoopcompatibility.mapred

Class HadoopMapFunction<KEYIN,VALUEIN,KEYOUT,VALUEOUT>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.api.common.functions.RichFlatMapFunction<Tuple2<KEYIN,VALUEIN>,Tuple2<KEYOUT,VALUEOUT>>

org.apache.flink.hadoopcompatibility.mapred.HadoopMapFunction<KEYIN,VALUEIN,KEYOUT,VALUEOUT>

All Implemented Interfaces:

Serializable, FlatMapFunction<Tuple2<KEYIN,VALUEIN>,Tuple2<KEYOUT,VALUEOUT>>, Function, RichFunction, ResultTypeQueryable<Tuple2<KEYOUT,VALUEOUT>>

@Public
public final class HadoopMapFunction<KEYIN,VALUEIN,KEYOUT,VALUEOUT>
extends RichFlatMapFunction<Tuple2<KEYIN,VALUEIN>,Tuple2<KEYOUT,VALUEOUT>>
implements ResultTypeQueryable<Tuple2<KEYOUT,VALUEOUT>>, Serializable

This wrapper maps a Hadoop Mapper (mapred API) to a Flink FlatMapFunction.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
HadoopMapFunction(org.apache.hadoop.mapred.Mapper<KEYIN,VALUEIN,KEYOUT,VALUEOUT> hadoopMapper) Maps a Hadoop Mapper (mapred API) to a Flink FlatMapFunction.
HadoopMapFunction(org.apache.hadoop.mapred.Mapper<KEYIN,VALUEIN,KEYOUT,VALUEOUT> hadoopMapper, org.apache.hadoop.mapred.JobConf conf) Maps a Hadoop Mapper (mapred API) to a Flink FlatMapFunction.

Method Summary

Modifier and Type	Method and Description
void	flatMap(Tuple2<KEYIN,VALUEIN> value, Collector<Tuple2<KEYOUT,VALUEOUT>> out) The core method of the FlatMapFunction.
TypeInformation<Tuple2<KEYOUT,VALUEOUT>>	getProducedType() Gets the data type (as a TypeInformation) produced by this function or input format.
void	open(Configuration parameters) Initialization method for the function.

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

close, getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopMapFunction

public HadoopMapFunction(org.apache.hadoop.mapred.Mapper<KEYIN,VALUEIN,KEYOUT,VALUEOUT> hadoopMapper)

Maps a Hadoop Mapper (mapred API) to a Flink FlatMapFunction.

Parameters:

hadoopMapper - The Hadoop Mapper to wrap.

HadoopMapFunction

public HadoopMapFunction(org.apache.hadoop.mapred.Mapper<KEYIN,VALUEIN,KEYOUT,VALUEOUT> hadoopMapper,
                         org.apache.hadoop.mapred.JobConf conf)

Maps a Hadoop Mapper (mapred API) to a Flink FlatMapFunction. The Hadoop Mapper is configured with the provided JobConf.

Parameters:

hadoopMapper - The Hadoop Mapper to wrap.

conf - The JobConf that is used to configure the Hadoop Mapper.

Method Detail

open

public void open(Configuration parameters)
          throws Exception

Description copied from interface: RichFunction

Initialization method for the function. It is called before the actual working methods (like map or join) and thus suitable for one time setup work. For functions that are part of an iteration, this method will be invoked at the beginning of each iteration superstep.

The configuration object passed to the function can be used for configuration and initialization. The configuration contains all parameters that were configured on the function in the program composition.

 public class MyFilter extends RichFilterFunction<String> {

     private String searchString;

     public void open(Configuration parameters) {
         this.searchString = parameters.getString("foo");
     }

     public boolean filter(String value) {
         return value.equals(searchString);
     }
 }
 

By default, this method does nothing.

Specified by:

open in interface RichFunction

Overrides:

open in class AbstractRichFunction

Parameters:

parameters - The configuration containing the parameters attached to the contract.

Throws:

Exception - Implementations may forward exceptions, which are caught by the runtime. When the runtime catches an exception, it aborts the task and lets the fail-over logic decide whether to retry the task execution.

See Also:

Configuration

flatMap

public void flatMap(Tuple2<KEYIN,VALUEIN> value,
                    Collector<Tuple2<KEYOUT,VALUEOUT>> out)
             throws Exception

Description copied from interface: FlatMapFunction

The core method of the FlatMapFunction. Takes an element from the input data set and transforms it into zero, one, or more elements.

Specified by:

flatMap in interface FlatMapFunction<Tuple2<KEYIN,VALUEIN>,Tuple2<KEYOUT,VALUEOUT>>

Specified by:

flatMap in class RichFlatMapFunction<Tuple2<KEYIN,VALUEIN>,Tuple2<KEYOUT,VALUEOUT>>

Parameters:

value - The input value.

out - The collector for returning result values.

Throws:

Exception - This method may throw exceptions. Throwing an exception will cause the operation to fail and may trigger recovery.

getProducedType

public TypeInformation<Tuple2<KEYOUT,VALUEOUT>> getProducedType()

Description copied from interface: ResultTypeQueryable

Gets the data type (as a TypeInformation) produced by this function or input format.

Specified by:

getProducedType in interface ResultTypeQueryable<Tuple2<KEYOUT,VALUEOUT>>

Returns:

The data type produced by this function or input format.