org.apache.flink.api.java.hadoop.mapred

Class HadoopOutputFormatBase<K,V,T>

java.lang.Object

org.apache.flink.api.common.io.RichOutputFormat<T>

org.apache.flink.api.java.hadoop.common.HadoopOutputFormatCommonBase<T>

org.apache.flink.api.java.hadoop.mapred.HadoopOutputFormatBase<K,V,T>

Type Parameters:

K - Type of Key

V - Type of Value

T - Record type.

All Implemented Interfaces:

Serializable, FinalizeOnMaster, OutputFormat<T>

Direct Known Subclasses:

HadoopOutputFormat

@Internal
public abstract class HadoopOutputFormatBase<K,V,T>
extends HadoopOutputFormatCommonBase<T>
implements FinalizeOnMaster

Common base for the mapred HadoopOutputFormat wrappers. There are implementations for Java and Scala.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.api.common.io.FinalizeOnMaster

FinalizeOnMaster.FinalizationContext

Nested classes/interfaces inherited from interface org.apache.flink.api.common.io.OutputFormat

OutputFormat.InitializationContext

Field Summary

Fields

Modifier and Type	Field and Description
protected static Object	CLOSE_MUTEX
protected static Object	CONFIGURE_MUTEX
protected org.apache.hadoop.mapred.TaskAttemptContext	context
protected org.apache.hadoop.mapred.JobConf	jobConf
protected org.apache.hadoop.mapred.OutputFormat<K,V>	mapredOutputFormat
protected static Object	OPEN_MUTEX
protected org.apache.hadoop.mapred.OutputCommitter	outputCommitter
protected org.apache.hadoop.mapred.RecordWriter<K,V>	recordWriter

Fields inherited from class org.apache.flink.api.java.hadoop.common.HadoopOutputFormatCommonBase

credentials

Constructor Summary

Constructors

Constructor and Description
HadoopOutputFormatBase(org.apache.hadoop.mapred.OutputFormat<K,V> mapredOutputFormat, org.apache.hadoop.mapred.JobConf job)

Method Summary

Modifier and Type	Method and Description
void	close() commit the task by moving the output file out from the temporary directory.
void	configure(Configuration parameters) Configures this output format.
void	finalizeGlobal(int parallelism) The method is invoked on the master (JobManager) after all (parallel) instances of an OutputFormat finished.
org.apache.hadoop.mapred.JobConf	getJobConf()
void	open(int taskNumber, int numTasks) create the temporary output file for hadoop RecordWriter.

Methods inherited from class org.apache.flink.api.java.hadoop.common.HadoopOutputFormatCommonBase

read, write

Methods inherited from class org.apache.flink.api.common.io.RichOutputFormat

getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.io.FinalizeOnMaster

finalizeGlobal

Methods inherited from interface org.apache.flink.api.common.io.OutputFormat

open, writeRecord

Field Detail

OPEN_MUTEX

protected static final Object OPEN_MUTEX

CONFIGURE_MUTEX

protected static final Object CONFIGURE_MUTEX

CLOSE_MUTEX

protected static final Object CLOSE_MUTEX

jobConf

protected org.apache.hadoop.mapred.JobConf jobConf

mapredOutputFormat

protected org.apache.hadoop.mapred.OutputFormat<K,V> mapredOutputFormat

recordWriter

protected transient org.apache.hadoop.mapred.RecordWriter<K,V> recordWriter

outputCommitter

protected transient org.apache.hadoop.mapred.OutputCommitter outputCommitter

context

protected transient org.apache.hadoop.mapred.TaskAttemptContext context

Constructor Detail

HadoopOutputFormatBase

public HadoopOutputFormatBase(org.apache.hadoop.mapred.OutputFormat<K,V> mapredOutputFormat,
                              org.apache.hadoop.mapred.JobConf job)

Method Detail

getJobConf

public org.apache.hadoop.mapred.JobConf getJobConf()

configure

public void configure(Configuration parameters)

Description copied from interface: OutputFormat

Configures this output format. Since output formats are instantiated generically and hence parameterless, this method is the place where the output formats set their basic fields based on configuration values.

This method is always called first on a newly instantiated output format.

Specified by:

configure in interface OutputFormat<T>

Parameters:

parameters - The configuration with all parameters.

open

public void open(int taskNumber,
                 int numTasks)
          throws IOException

create the temporary output file for hadoop RecordWriter.

Specified by:

open in interface OutputFormat<T>

Parameters:

taskNumber - The number of the parallel instance.

numTasks - The number of parallel tasks.

Throws:

IOException

close

public void close()
           throws IOException

commit the task by moving the output file out from the temporary directory.

Specified by:

close in interface OutputFormat<T>

Throws:

IOException

finalizeGlobal

public void finalizeGlobal(int parallelism)
                    throws IOException

Description copied from interface: FinalizeOnMaster

The method is invoked on the master (JobManager) after all (parallel) instances of an OutputFormat finished.

Specified by:

finalizeGlobal in interface FinalizeOnMaster

Parameters:

parallelism - The parallelism with which the format or functions was run.

Throws:

IOException - The finalization may throw exceptions, which may cause the job to abort.