org.apache.flink.api.java.hadoop.mapreduce

Class HadoopOutputFormatBase<K,V,T>

java.lang.Object

org.apache.flink.api.common.io.RichOutputFormat<T>

org.apache.flink.api.java.hadoop.common.HadoopOutputFormatCommonBase<T>

org.apache.flink.api.java.hadoop.mapreduce.HadoopOutputFormatBase<K,V,T>

All Implemented Interfaces:

Serializable, FinalizeOnMaster, OutputFormat<T>

Direct Known Subclasses:

HadoopOutputFormat

@Internal
public abstract class HadoopOutputFormatBase<K,V,T>
extends HadoopOutputFormatCommonBase<T>
implements FinalizeOnMaster

Base class shared between the Java and Scala API of Flink.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.api.common.io.FinalizeOnMaster

FinalizeOnMaster.FinalizationContext

Nested classes/interfaces inherited from interface org.apache.flink.api.common.io.OutputFormat

OutputFormat.InitializationContext

Field Summary

Fields

Modifier and Type	Field and Description
protected static Object	CLOSE_MUTEX
protected org.apache.hadoop.conf.Configuration	configuration
protected static Object	CONFIGURE_MUTEX
protected org.apache.hadoop.mapreduce.TaskAttemptContext	context
protected org.apache.hadoop.mapreduce.OutputFormat<K,V>	mapreduceOutputFormat
protected static Object	OPEN_MUTEX
protected org.apache.hadoop.mapreduce.OutputCommitter	outputCommitter
protected org.apache.hadoop.mapreduce.RecordWriter<K,V>	recordWriter
protected int	taskNumber

Fields inherited from class org.apache.flink.api.java.hadoop.common.HadoopOutputFormatCommonBase

credentials

Constructor Summary

Constructors

Constructor and Description
HadoopOutputFormatBase(org.apache.hadoop.mapreduce.OutputFormat<K,V> mapreduceOutputFormat, org.apache.hadoop.mapreduce.Job job)

Method Summary

Modifier and Type	Method and Description
void	close() commit the task by moving the output file out from the temporary directory.
void	configure(Configuration parameters) Configures this output format.
void	finalizeGlobal(int parallelism) The method is invoked on the master (JobManager) after all (parallel) instances of an OutputFormat finished.
org.apache.hadoop.conf.Configuration	getConfiguration()
void	open(int taskNumber, int numTasks) create the temporary output file for hadoop RecordWriter.

Methods inherited from class org.apache.flink.api.java.hadoop.common.HadoopOutputFormatCommonBase

read, write

Methods inherited from class org.apache.flink.api.common.io.RichOutputFormat

getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.io.FinalizeOnMaster

finalizeGlobal

Methods inherited from interface org.apache.flink.api.common.io.OutputFormat

open, writeRecord

Field Detail

OPEN_MUTEX

protected static final Object OPEN_MUTEX

CONFIGURE_MUTEX

protected static final Object CONFIGURE_MUTEX

CLOSE_MUTEX

protected static final Object CLOSE_MUTEX

configuration

protected org.apache.hadoop.conf.Configuration configuration

mapreduceOutputFormat

protected org.apache.hadoop.mapreduce.OutputFormat<K,V> mapreduceOutputFormat

recordWriter

protected transient org.apache.hadoop.mapreduce.RecordWriter<K,V> recordWriter

outputCommitter

protected transient org.apache.hadoop.mapreduce.OutputCommitter outputCommitter

context

protected transient org.apache.hadoop.mapreduce.TaskAttemptContext context

taskNumber

protected transient int taskNumber

Constructor Detail

HadoopOutputFormatBase

public HadoopOutputFormatBase(org.apache.hadoop.mapreduce.OutputFormat<K,V> mapreduceOutputFormat,
                              org.apache.hadoop.mapreduce.Job job)

Method Detail

getConfiguration

public org.apache.hadoop.conf.Configuration getConfiguration()

configure

public void configure(Configuration parameters)

Description copied from interface: OutputFormat

Configures this output format. Since output formats are instantiated generically and hence parameterless, this method is the place where the output formats set their basic fields based on configuration values.

This method is always called first on a newly instantiated output format.

Specified by:

configure in interface OutputFormat<T>

Parameters:

parameters - The configuration with all parameters.

open

public void open(int taskNumber,
                 int numTasks)
          throws IOException

create the temporary output file for hadoop RecordWriter.

Specified by:

open in interface OutputFormat<T>

Parameters:

taskNumber - The number of the parallel instance.

numTasks - The number of parallel tasks.

Throws:

IOException

close

public void close()
           throws IOException

commit the task by moving the output file out from the temporary directory.

Specified by:

close in interface OutputFormat<T>

Throws:

IOException

finalizeGlobal

public void finalizeGlobal(int parallelism)
                    throws IOException

Description copied from interface: FinalizeOnMaster

The method is invoked on the master (JobManager) after all (parallel) instances of an OutputFormat finished.

Specified by:

finalizeGlobal in interface FinalizeOnMaster

Parameters:

parallelism - The parallelism with which the format or functions was run.

Throws:

IOException - The finalization may throw exceptions, which may cause the job to abort.