org.apache.flink.formats.parquet

Class ParquetColumnarRowInputFormat<SplitT extends FileSourceSplit>

java.lang.Object

org.apache.flink.formats.parquet.ParquetVectorizedInputFormat<RowData,SplitT>

org.apache.flink.formats.parquet.ParquetColumnarRowInputFormat<SplitT>

All Implemented Interfaces:

Serializable, ResultTypeQueryable<RowData>, BulkFormat<RowData,SplitT>, FileBasedStatisticsReportableInputFormat

public class ParquetColumnarRowInputFormat<SplitT extends FileSourceSplit>
extends ParquetVectorizedInputFormat<RowData,SplitT>
implements FileBasedStatisticsReportableInputFormat

A ParquetVectorizedInputFormat to provide RowData iterator. Using ColumnarRowData to provide a row view of column batch.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.formats.parquet.ParquetVectorizedInputFormat

ParquetVectorizedInputFormat.ParquetReaderBatch<T>

Nested classes/interfaces inherited from interface org.apache.flink.connector.file.src.reader.BulkFormat

BulkFormat.Reader<T>, BulkFormat.RecordIterator<T>

Field Summary

Fields inherited from class org.apache.flink.formats.parquet.ParquetVectorizedInputFormat

hadoopConfig, isUtcTimestamp

Constructor Summary

Constructors

Constructor and Description
ParquetColumnarRowInputFormat(org.apache.hadoop.conf.Configuration hadoopConfig, RowType projectedType, TypeInformation<RowData> producedTypeInfo, int batchSize, boolean isUtcTimestamp, boolean isCaseSensitive) Constructor to create parquet format without extra fields.

Method Summary

Modifier and Type	Method and Description
static <SplitT extends FileSourceSplit> ParquetColumnarRowInputFormat<SplitT>	createPartitionedFormat(org.apache.hadoop.conf.Configuration hadoopConfig, RowType producedRowType, TypeInformation<RowData> producedTypeInfo, List<String> partitionKeys, PartitionFieldExtractor<SplitT> extractor, int batchSize, boolean isUtcTimestamp, boolean isCaseSensitive) Create a partitioned ParquetColumnarRowInputFormat, the partition columns can be generated by Path.
protected ParquetVectorizedInputFormat.ParquetReaderBatch<RowData>	createReaderBatch(WritableColumnVector[] writableVectors, VectorizedColumnBatch columnarBatch, Pool.Recycler<ParquetVectorizedInputFormat.ParquetReaderBatch<RowData>> recycler)
TypeInformation<RowData>	getProducedType() Gets the type produced by this format.
protected int	numBatchesToCirculate(Configuration config)
TableStats	reportStatistics(List<Path> files, DataType producedDataType) Returns the estimated statistics of this input format.

Methods inherited from class org.apache.flink.formats.parquet.ParquetVectorizedInputFormat

createReader, isSplittable, restoreReader

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ParquetColumnarRowInputFormat

public ParquetColumnarRowInputFormat(org.apache.hadoop.conf.Configuration hadoopConfig,
                                     RowType projectedType,
                                     TypeInformation<RowData> producedTypeInfo,
                                     int batchSize,
                                     boolean isUtcTimestamp,
                                     boolean isCaseSensitive)

Constructor to create parquet format without extra fields.

Method Detail

numBatchesToCirculate

protected int numBatchesToCirculate(Configuration config)

Overrides:

numBatchesToCirculate in class ParquetVectorizedInputFormat<RowData,SplitT extends FileSourceSplit>

createReaderBatch

protected ParquetVectorizedInputFormat.ParquetReaderBatch<RowData> createReaderBatch(WritableColumnVector[] writableVectors,
                                                                                     VectorizedColumnBatch columnarBatch,
                                                                                     Pool.Recycler<ParquetVectorizedInputFormat.ParquetReaderBatch<RowData>> recycler)

Specified by:

createReaderBatch in class ParquetVectorizedInputFormat<RowData,SplitT extends FileSourceSplit>

Parameters:

writableVectors - vectors to be write

columnarBatch - vectors to be read

recycler - batch recycler

getProducedType

public TypeInformation<RowData> getProducedType()

Description copied from interface: BulkFormat

Gets the type produced by this format. This type will be the type produced by the file source as a whole.

Specified by:

getProducedType in interface ResultTypeQueryable<RowData>

Specified by:

getProducedType in interface BulkFormat<RowData,SplitT extends FileSourceSplit>

Returns:

The data type produced by this function or input format.

reportStatistics

public TableStats reportStatistics(List<Path> files,
                                   DataType producedDataType)

Description copied from interface: FileBasedStatisticsReportableInputFormat

Returns the estimated statistics of this input format.

Specified by:

reportStatistics in interface FileBasedStatisticsReportableInputFormat

Parameters:

files - The files to be estimated.

producedDataType - the final output type of the format.

createPartitionedFormat

public static <SplitT extends FileSourceSplit> ParquetColumnarRowInputFormat<SplitT> createPartitionedFormat(org.apache.hadoop.conf.Configuration hadoopConfig,
                                                                                                             RowType producedRowType,
                                                                                                             TypeInformation<RowData> producedTypeInfo,
                                                                                                             List<String> partitionKeys,
                                                                                                             PartitionFieldExtractor<SplitT> extractor,
                                                                                                             int batchSize,
                                                                                                             boolean isUtcTimestamp,
                                                                                                             boolean isCaseSensitive)

Create a partitioned ParquetColumnarRowInputFormat, the partition columns can be generated by Path.