org.apache.flink.orc

Class OrcColumnarRowFileInputFormat<BatchT,SplitT extends FileSourceSplit>

java.lang.Object

org.apache.flink.orc.AbstractOrcFileInputFormat<RowData,BatchT,SplitT>

org.apache.flink.orc.OrcColumnarRowFileInputFormat<BatchT,SplitT>

All Implemented Interfaces:

Serializable, ResultTypeQueryable<RowData>, BulkFormat<RowData,SplitT>

public class OrcColumnarRowFileInputFormat<BatchT,SplitT extends FileSourceSplit>
extends AbstractOrcFileInputFormat<RowData,BatchT,SplitT>

An ORC reader that produces a stream of ColumnarRowData records.

This class can add extra fields through ColumnBatchFactory, for example, add partition fields, which can be extracted from path. Therefore, the getProducedType() may be different and types of extra fields need to be added.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.orc.AbstractOrcFileInputFormat

AbstractOrcFileInputFormat.OrcReaderBatch<T,BatchT>, AbstractOrcFileInputFormat.OrcVectorizedReader<T,BatchT>

Nested classes/interfaces inherited from interface org.apache.flink.connector.file.src.reader.BulkFormat

BulkFormat.Reader<T>, BulkFormat.RecordIterator<T>

Field Summary

Fields inherited from class org.apache.flink.orc.AbstractOrcFileInputFormat

batchSize, conjunctPredicates, hadoopConfigWrapper, schema, selectedFields, shim

Constructor Summary

Constructors

Constructor and Description
OrcColumnarRowFileInputFormat(OrcShim<BatchT> shim, Configuration hadoopConfig, org.apache.orc.TypeDescription schema, int[] selectedFields, List<OrcFilters.Predicate> conjunctPredicates, int batchSize, ColumnBatchFactory<BatchT,SplitT> batchFactory, RowType projectedOutputType)

Method Summary

Modifier and Type	Method and Description
static <SplitT extends FileSourceSplit> OrcColumnarRowFileInputFormat<org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch,SplitT>	createPartitionedFormat(OrcShim<org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch> shim, Configuration hadoopConfig, RowType tableType, List<String> partitionKeys, PartitionFieldExtractor<SplitT> extractor, int[] selectedFields, List<OrcFilters.Predicate> conjunctPredicates, int batchSize) Create a partitioned OrcColumnarRowFileInputFormat, the partition columns can be generated by split.
AbstractOrcFileInputFormat.OrcReaderBatch<RowData,BatchT>	createReaderBatch(SplitT split, OrcVectorizedBatchWrapper<BatchT> orcBatch, Pool.Recycler<AbstractOrcFileInputFormat.OrcReaderBatch<RowData,BatchT>> recycler, int batchSize) Creates the AbstractOrcFileInputFormat.OrcReaderBatch structure, which is responsible for holding the data structures that hold the batch data (column vectors, row arrays, ...) and the batch conversion from the ORC representation to the result format.
TypeInformation<RowData>	getProducedType() Gets the type produced by this format.

Methods inherited from class org.apache.flink.orc.AbstractOrcFileInputFormat

createReader, isSplittable, restoreReader

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

OrcColumnarRowFileInputFormat

public OrcColumnarRowFileInputFormat(OrcShim<BatchT> shim,
                                     Configuration hadoopConfig,
                                     org.apache.orc.TypeDescription schema,
                                     int[] selectedFields,
                                     List<OrcFilters.Predicate> conjunctPredicates,
                                     int batchSize,
                                     ColumnBatchFactory<BatchT,SplitT> batchFactory,
                                     RowType projectedOutputType)

Method Detail

createReaderBatch

public AbstractOrcFileInputFormat.OrcReaderBatch<RowData,BatchT> createReaderBatch(SplitT split,
                                                                                   OrcVectorizedBatchWrapper<BatchT> orcBatch,
                                                                                   Pool.Recycler<AbstractOrcFileInputFormat.OrcReaderBatch<RowData,BatchT>> recycler,
                                                                                   int batchSize)

Description copied from class: AbstractOrcFileInputFormat

Creates the AbstractOrcFileInputFormat.OrcReaderBatch structure, which is responsible for holding the data structures that hold the batch data (column vectors, row arrays, ...) and the batch conversion from the ORC representation to the result format.

Specified by:

createReaderBatch in class AbstractOrcFileInputFormat<RowData,BatchT,SplitT extends FileSourceSplit>

getProducedType

public TypeInformation<RowData> getProducedType()

Description copied from class: AbstractOrcFileInputFormat

Gets the type produced by this format.

Specified by:

getProducedType in interface ResultTypeQueryable<RowData>

Specified by:

getProducedType in interface BulkFormat<RowData,SplitT extends FileSourceSplit>

Specified by:

getProducedType in class AbstractOrcFileInputFormat<RowData,BatchT,SplitT extends FileSourceSplit>

Returns:

The data type produced by this function or input format.

createPartitionedFormat

public static <SplitT extends FileSourceSplit> OrcColumnarRowFileInputFormat<org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch,SplitT> createPartitionedFormat(OrcShim<org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch> shim,
                                                                                                                                                                      Configuration hadoopConfig,
                                                                                                                                                                      RowType tableType,
                                                                                                                                                                      List<String> partitionKeys,
                                                                                                                                                                      PartitionFieldExtractor<SplitT> extractor,
                                                                                                                                                                      int[] selectedFields,
                                                                                                                                                                      List<OrcFilters.Predicate> conjunctPredicates,
                                                                                                                                                                      int batchSize)

Create a partitioned OrcColumnarRowFileInputFormat, the partition columns can be generated by split.