org.apache.flink.formats.json

Class JsonFileSystemFormatFactory.JsonInputFormat

java.lang.Object

org.apache.flink.api.common.io.RichInputFormat<OT,FileInputSplit>

org.apache.flink.api.common.io.FileInputFormat<OT>

org.apache.flink.api.common.io.DelimitedInputFormat<RowData>

org.apache.flink.formats.json.JsonFileSystemFormatFactory.JsonInputFormat

All Implemented Interfaces:

Serializable, CheckpointableInputFormat<FileInputSplit,Long>, InputFormat<RowData,FileInputSplit>, InputSplitSource<FileInputSplit>

Enclosing class:

JsonFileSystemFormatFactory

public static class JsonFileSystemFormatFactory.JsonInputFormat
extends DelimitedInputFormat<RowData>

A JsonFileSystemFormatFactory.JsonInputFormat is responsible to read RowData records from json format files.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.api.common.io.FileInputFormat

FileInputFormat.FileBaseStatistics, FileInputFormat.InputSplitOpenThread

Field Summary

Fields inherited from class org.apache.flink.api.common.io.DelimitedInputFormat

currBuffer, currLen, currOffset, RECORD_DELIMITER

Fields inherited from class org.apache.flink.api.common.io.FileInputFormat

currentSplit, ENUMERATE_NESTED_FILES_FLAG, enumerateNestedFiles, filePath, INFLATER_INPUT_STREAM_FACTORIES, minSplitSize, numSplits, openTimeout, READ_WHOLE_SPLIT_FLAG, splitLength, splitStart, stream, unsplittable

Constructor Summary

Constructors

Constructor and Description
JsonInputFormat(Path[] filePaths, DataType[] fieldTypes, String[] fieldNames, int[] selectFields, List<String> partitionKeys, String defaultPartValue, long limit, int[] jsonSelectFieldToProjectFieldMapping, int[] jsonSelectFieldToJsonFieldMapping, JsonRowDataDeserializationSchema deserializationSchema)

Method Summary

Modifier and Type	Method and Description
RowData	nextRecord(RowData record) Reads the next record from the input.
void	open(FileInputSplit split) Opens the given input split.
boolean	reachedEnd() Checks whether the current split is at its end.
RowData	readRecord(RowData reuse, byte[] bytes, int offset, int numBytes) This function parses the given byte array which represents a serialized record.
boolean	supportsMultiPaths() Override this method to supports multiple paths.

Methods inherited from class org.apache.flink.api.common.io.DelimitedInputFormat

close, configure, getBufferSize, getCharset, getCurrentState, getDelimiter, getLineLengthLimit, getNumLineSamples, getStatistics, initializeSplit, loadConfigParameters, loadGlobalConfigParams, readLine, reopen, setBufferSize, setCharset, setDelimiter, setDelimiter, setDelimiter, setLineLengthLimit, setNumLineSamples

Methods inherited from class org.apache.flink.api.common.io.FileInputFormat

acceptFile, createInputSplits, decorateInputStream, extractFileExtension, getFilePath, getFilePaths, getFileStats, getFileStats, getInflaterInputStreamFactory, getInputSplitAssigner, getMinSplitSize, getNestedFileEnumeration, getNumSplits, getOpenTimeout, getSplitLength, getSplitStart, registerInflaterInputStreamFactory, setFilePath, setFilePath, setFilePaths, setFilePaths, setFilesFilter, setMinSplitSize, setNestedFileEnumeration, setNumSplits, setOpenTimeout, testForUnsplittable, toString

Methods inherited from class org.apache.flink.api.common.io.RichInputFormat

closeInputFormat, getRuntimeContext, openInputFormat, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

JsonInputFormat

public JsonInputFormat(Path[] filePaths,
                       DataType[] fieldTypes,
                       String[] fieldNames,
                       int[] selectFields,
                       List<String> partitionKeys,
                       String defaultPartValue,
                       long limit,
                       int[] jsonSelectFieldToProjectFieldMapping,
                       int[] jsonSelectFieldToJsonFieldMapping,
                       JsonRowDataDeserializationSchema deserializationSchema)

Method Detail

supportsMultiPaths

public boolean supportsMultiPaths()

Description copied from class: FileInputFormat

Override this method to supports multiple paths. When this method will be removed, all FileInputFormats have to support multiple paths.

Overrides:

supportsMultiPaths in class FileInputFormat<RowData>

Returns:

True if the FileInputFormat supports multiple paths, false otherwise.

open

public void open(FileInputSplit split)
          throws IOException

Description copied from class: DelimitedInputFormat

Opens the given input split. This method opens the input stream to the specified file, allocates read buffers and positions the stream at the correct position, making sure that any partial record at the beginning is skipped.

Specified by:

open in interface InputFormat<RowData,FileInputSplit>

Overrides:

open in class DelimitedInputFormat<RowData>

Parameters:

split - The input split to open.

Throws:

IOException - Thrown, if the spit could not be opened due to an I/O problem.

See Also:

FileInputFormat.open(org.apache.flink.core.fs.FileInputSplit)

reachedEnd

public boolean reachedEnd()

Description copied from class: DelimitedInputFormat

Checks whether the current split is at its end.

Specified by:

reachedEnd in interface InputFormat<RowData,FileInputSplit>

Overrides:

reachedEnd in class DelimitedInputFormat<RowData>

Returns:

True, if the split is at its end, false otherwise.

readRecord

public RowData readRecord(RowData reuse,
                          byte[] bytes,
                          int offset,
                          int numBytes)
                   throws IOException

Description copied from class: DelimitedInputFormat

This function parses the given byte array which represents a serialized record. The function returns a valid record or throws an IOException.

Specified by:

readRecord in class DelimitedInputFormat<RowData>

Parameters:

reuse - An optionally reusable object.

bytes - Binary data of serialized records.

offset - The offset where to start to read the record data.

numBytes - The number of bytes that can be read starting at the offset position.

Returns:

Returns the read record if it was successfully deserialized.

Throws:

IOException - if the record could not be read.

nextRecord

public RowData nextRecord(RowData record)
                   throws IOException

Description copied from interface: InputFormat

Reads the next record from the input.

When this method is called, the input format it guaranteed to be opened.

Specified by:

nextRecord in interface InputFormat<RowData,FileInputSplit>

Overrides:

nextRecord in class DelimitedInputFormat<RowData>

Parameters:

record - Object that may be reused.

Returns:

Read record.

Throws:

IOException - Thrown, if an I/O error occurred.