org.apache.flink.formats.parquet

Class ParquetPojoInputFormat<E>

java.lang.Object

org.apache.flink.api.common.io.RichInputFormat<OT,FileInputSplit>

org.apache.flink.api.common.io.FileInputFormat<E>

org.apache.flink.formats.parquet.ParquetInputFormat<E>

org.apache.flink.formats.parquet.ParquetPojoInputFormat<E>

All Implemented Interfaces:

Serializable, CheckpointableInputFormat<FileInputSplit,Tuple2<Long,Long>>, InputFormat<E,FileInputSplit>, InputSplitSource<FileInputSplit>

public class ParquetPojoInputFormat<E>
extends ParquetInputFormat<E>

An implementation of ParquetInputFormat to read POJO records from Parquet files.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.api.common.io.FileInputFormat

FileInputFormat.FileBaseStatistics, FileInputFormat.InputSplitOpenThread

Field Summary

Fields inherited from class org.apache.flink.formats.parquet.ParquetInputFormat

PARQUET_SKIP_CORRUPTED_RECORD, PARQUET_SKIP_WRONG_SCHEMA_SPLITS

Fields inherited from class org.apache.flink.api.common.io.FileInputFormat

currentSplit, ENUMERATE_NESTED_FILES_FLAG, enumerateNestedFiles, filePath, INFLATER_INPUT_STREAM_FACTORIES, minSplitSize, numSplits, openTimeout, READ_WHOLE_SPLIT_FLAG, splitLength, splitStart, stream, unsplittable

Constructor Summary

Constructors

Constructor and Description
ParquetPojoInputFormat(Path filePath, org.apache.parquet.schema.MessageType messageType, PojoTypeInfo<E> pojoTypeInfo)

Method Summary

Modifier and Type	Method and Description
protected E	convert(Row row) This ParquetInputFormat read parquet record as Row by default.
void	open(FileInputSplit split) Opens an input stream to the file defined in the input format.

Methods inherited from class org.apache.flink.formats.parquet.ParquetInputFormat

close, configure, getCurrentState, getFieldNames, getFieldTypes, getPredicate, nextRecord, reachedEnd, reopen, selectFields, setFilterPredicate

Methods inherited from class org.apache.flink.api.common.io.FileInputFormat

acceptFile, createInputSplits, decorateInputStream, extractFileExtension, getFilePath, getFilePaths, getFileStats, getFileStats, getInflaterInputStreamFactory, getInputSplitAssigner, getMinSplitSize, getNestedFileEnumeration, getNumSplits, getOpenTimeout, getSplitLength, getSplitStart, getStatistics, registerInflaterInputStreamFactory, setFilePath, setFilePath, setFilePaths, setFilePaths, setFilesFilter, setMinSplitSize, setNestedFileEnumeration, setNumSplits, setOpenTimeout, supportsMultiPaths, testForUnsplittable, toString

Methods inherited from class org.apache.flink.api.common.io.RichInputFormat

closeInputFormat, getRuntimeContext, openInputFormat, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

ParquetPojoInputFormat

public ParquetPojoInputFormat(Path filePath,
                              org.apache.parquet.schema.MessageType messageType,
                              PojoTypeInfo<E> pojoTypeInfo)

Method Detail

open

public void open(FileInputSplit split)
          throws IOException

Description copied from class: FileInputFormat

Opens an input stream to the file defined in the input format. The stream is positioned at the beginning of the given split.

The stream is actually opened in an asynchronous thread to make sure any interruptions to the thread working on the input format do not reach the file system.

Specified by:

open in interface InputFormat<E,FileInputSplit>

Overrides:

open in class ParquetInputFormat<E>

Parameters:

split - The split to be opened.

Throws:

IOException - Thrown, if the spit could not be opened due to an I/O problem.

convert

protected E convert(Row row)

Description copied from class: ParquetInputFormat

This ParquetInputFormat read parquet record as Row by default. Sub classes of it can extend this method to further convert row to other types, such as POJO, Map or Tuple.

Specified by:

convert in class ParquetInputFormat<E>

Parameters:

row - row read from parquet file

Returns:

E target result type