org.apache.flink.orc

Class OrcInputFormat<T>

java.lang.Object

org.apache.flink.api.common.io.RichInputFormat<OT,FileInputSplit>

org.apache.flink.api.common.io.FileInputFormat<T>

org.apache.flink.orc.OrcInputFormat<T>

All Implemented Interfaces:

Serializable, InputFormat<T,FileInputSplit>, InputSplitSource<FileInputSplit>

Direct Known Subclasses:

OrcRowInputFormat

public abstract class OrcInputFormat<T>
extends FileInputFormat<T>

InputFormat to read ORC files.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.api.common.io.FileInputFormat

FileInputFormat.FileBaseStatistics, FileInputFormat.InputSplitOpenThread

Field Summary

Fields

Modifier and Type	Field and Description
protected int	batchSize
protected Configuration	conf
protected ArrayList<OrcSplitReader.Predicate>	conjunctPredicates
protected OrcSplitReader<T>	reader
protected org.apache.orc.TypeDescription	schema
protected int[]	selectedFields

Fields inherited from class org.apache.flink.api.common.io.FileInputFormat

currentSplit, ENUMERATE_NESTED_FILES_FLAG, enumerateNestedFiles, filePath, INFLATER_INPUT_STREAM_FACTORIES, minSplitSize, numSplits, openTimeout, READ_WHOLE_SPLIT_FLAG, splitLength, splitStart, stream, unsplittable

Constructor Summary

Constructors

Constructor and Description
OrcInputFormat(Path path, org.apache.orc.TypeDescription orcSchema, Configuration orcConfig, int batchSize) Creates an OrcInputFormat.

Method Summary

Modifier and Type	Method and Description
void	addPredicate(OrcSplitReader.Predicate predicate) Adds a filter predicate to reduce the number of rows to be returned by the input format.
void	close() Closes the file input stream of the input format.
void	closeInputFormat() Closes this InputFormat instance.
T	nextRecord(T reuse) Reads the next record from the input.
boolean	reachedEnd() Method used to check if the end of the input is reached.
void	selectFields(int... selectedFields) Selects the fields from the ORC schema that are returned by InputFormat.
boolean	supportsMultiPaths() Override this method to supports multiple paths.

Methods inherited from class org.apache.flink.api.common.io.FileInputFormat

acceptFile, configure, createInputSplits, decorateInputStream, extractFileExtension, getFilePath, getFilePaths, getFileStats, getFileStats, getInflaterInputStreamFactory, getInputSplitAssigner, getMinSplitSize, getNestedFileEnumeration, getNumSplits, getOpenTimeout, getSplitLength, getSplitStart, getStatistics, open, registerInflaterInputStreamFactory, setFilePath, setFilePath, setFilePaths, setFilePaths, setFilesFilter, setMinSplitSize, setNestedFileEnumeration, setNumSplits, setOpenTimeout, testForUnsplittable, toString

Methods inherited from class org.apache.flink.api.common.io.RichInputFormat

getRuntimeContext, openInputFormat, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

batchSize

protected int batchSize

conf

protected Configuration conf

schema

protected org.apache.orc.TypeDescription schema

selectedFields

protected int[] selectedFields

conjunctPredicates

protected ArrayList<OrcSplitReader.Predicate> conjunctPredicates

reader

protected transient OrcSplitReader<T> reader

Constructor Detail

OrcInputFormat

public OrcInputFormat(Path path,
                      org.apache.orc.TypeDescription orcSchema,
                      Configuration orcConfig,
                      int batchSize)

Creates an OrcInputFormat.

Parameters:

path - The path to read ORC files from.

orcSchema - The schema of the ORC files as ORC TypeDescription.

orcConfig - The configuration to read the ORC files with.

batchSize - The number of Row objects to read in a batch.

Method Detail

selectFields

public void selectFields(int... selectedFields)

Selects the fields from the ORC schema that are returned by InputFormat.

Parameters:

selectedFields - The indices of the fields of the ORC schema that are returned by the InputFormat.

addPredicate

public void addPredicate(OrcSplitReader.Predicate predicate)

Adds a filter predicate to reduce the number of rows to be returned by the input format. Multiple conjunctive predicates can be added by calling this method multiple times.

Note: Predicates can significantly reduce the amount of data that is read. However, the OrcInputFormat does not guarantee that all returned rows qualify the predicates. Moreover, predicates are only applied if the referenced field is among the selected fields.

Parameters:

predicate - The filter predicate.

close

public void close()
           throws IOException

Description copied from class: FileInputFormat

Closes the file input stream of the input format.

Specified by:

close in interface InputFormat<T,FileInputSplit>

Overrides:

close in class FileInputFormat<T>

Throws:

IOException - Thrown, if the input could not be closed properly.

closeInputFormat

public void closeInputFormat()
                      throws IOException

Description copied from class: RichInputFormat

Closes this InputFormat instance. This method is called once per parallel instance. Resources allocated during RichInputFormat.openInputFormat() should be closed in this method.

Overrides:

closeInputFormat in class RichInputFormat<T,FileInputSplit>

Throws:

IOException - in case closing the resources failed

See Also:

InputFormat

reachedEnd

public boolean reachedEnd()
                   throws IOException

Description copied from interface: InputFormat

Method used to check if the end of the input is reached.

When this method is called, the input format it guaranteed to be opened.

Returns:

True if the end is reached, otherwise false.

Throws:

IOException - Thrown, if an I/O error occurred.

nextRecord

public T nextRecord(T reuse)
             throws IOException

Description copied from interface: InputFormat

Reads the next record from the input.

When this method is called, the input format it guaranteed to be opened.

Parameters:

reuse - Object that may be reused.

Returns:

Read record.

Throws:

IOException - Thrown, if an I/O error occurred.

supportsMultiPaths

public boolean supportsMultiPaths()

Description copied from class: FileInputFormat

Override this method to supports multiple paths. When this method will be removed, all FileInputFormats have to support multiple paths.

Overrides:

supportsMultiPaths in class FileInputFormat<T>

Returns:

True if the FileInputFormat supports multiple paths, false otherwise.