This documentation is for an out-of-date version of Apache Flink. We recommend you use the latest stable version.
Hadoop formats #
Project Configuration #
Support for Hadoop is contained in the flink-hadoop-compatibility
Maven module.
Add the following dependency to your pom.xml
to use hadoop
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-hadoop-compatibility_2.11</artifactId>
<version>1.14.4</version>
</dependency>
If you want to run your Flink application locally (e.g. from your IDE), you also need to add
a hadoop-client
dependency such as:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.8.3</version>
<scope>provided</scope>
</dependency>
Using Hadoop InputFormats #
To use Hadoop InputFormats
with Flink the format must first be wrapped
using either readHadoopFile
or createHadoopInput
of the
HadoopInputs
utility class.
The former is used for input formats derived
from FileInputFormat
while the latter has to be used for general purpose
input formats.
The resulting InputFormat
can be used to create a data source by using
ExecutionEnvironmen#createInput
.
The resulting DataStream
contains 2-tuples where the first field
is the key and the second field is the value retrieved from the Hadoop
InputFormat.
The following example shows how to use Hadoop’s TextInputFormat
.
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
DataStream<Tuple2<LongWritable, Text>> input =
env.createInput(HadoopInputs.readHadoopFile(new TextInputFormat(),
LongWritable.class, Text.class, textPath));
// Do something with the data.
[...]
val env = StreamExecutionEnvironment.getExecutionEnvironment
val input: DataStream[(LongWritable, Text)] =
env.createInput(HadoopInputs.readHadoopFile(
new TextInputFormat, classOf[LongWritable], classOf[Text], textPath))
// Do something with the data.
[...]
Using Hadoop OutputFormats #
Flink provides a compatibility wrapper for Hadoop OutputFormats
. Any class
that implements org.apache.hadoop.mapred.OutputFormat
or extends
org.apache.hadoop.mapreduce.OutputFormat
is supported.
The OutputFormat wrapper expects its input data to be a DataSet containing
2-tuples of key and value. These are to be processed by the Hadoop OutputFormat.
The following example shows how to use Hadoop’s TextOutputFormat
.
// Obtain the result we want to emit
DataStream<Tuple2<Text, IntWritable>> hadoopResult = [...]
// Set up the Hadoop TextOutputFormat.
HadoopOutputFormat<Text, IntWritable> hadoopOF =
// create the Flink wrapper.
new HadoopOutputFormat<Text, IntWritable>(
// set the Hadoop OutputFormat and specify the job.
new TextOutputFormat<Text, IntWritable>(), job
);
hadoopOF.getConfiguration().set("mapreduce.output.textoutputformat.separator", " ");
TextOutputFormat.setOutputPath(job, new Path(outputPath));
// Emit data using the Hadoop TextOutputFormat.
hadoopResult.output(hadoopOF);
// Obtain your result to emit.
val hadoopResult: DataStream[(Text, IntWritable)] = [...]
val hadoopOF = new HadoopOutputFormat[Text,IntWritable](
new TextOutputFormat[Text, IntWritable],
new JobConf)
hadoopOF.getJobConf.set("mapred.textoutputformat.separator", " ")
FileOutputFormat.setOutputPath(hadoopOF.getJobConf, new Path(resultPath))
hadoopResult.output(hadoopOF)