org.apache.flink.formats.csv

Class RowCsvInputFormat

java.lang.Object

org.apache.flink.api.common.io.RichInputFormat<OT,FileInputSplit>

org.apache.flink.api.common.io.FileInputFormat<T>

org.apache.flink.formats.csv.AbstractCsvInputFormat<Row>

org.apache.flink.formats.csv.RowCsvInputFormat

All Implemented Interfaces:

Serializable, InputFormat<Row,FileInputSplit>, InputSplitSource<FileInputSplit>

public class RowCsvInputFormat
extends AbstractCsvInputFormat<Row>

Input format that reads csv into Row.

Different from old csv org.apache.flink.api.java.io.RowCsvInputFormat: 1.New csv will emit this row (Fill null the remaining fields) when row is too short. But Old csv will skip this too short row. 2.New csv, escape char will be removed. But old csv will keep the escape char.

These can be continuously improved in new csv input format: 1.New csv not support configure comment char. The comment char is "#". 2.New csv not support configure multi chars field delimiter. 3.New csv not support read first N, it will throw exception. 4.Only support configure line delimiter: "\r" or "\n" or "\r\n".

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	RowCsvInputFormat.Builder A builder for creating a RowCsvInputFormat.

Nested classes/interfaces inherited from class org.apache.flink.api.common.io.FileInputFormat

FileInputFormat.FileBaseStatistics, FileInputFormat.InputSplitOpenThread

Field Summary

Fields inherited from class org.apache.flink.formats.csv.AbstractCsvInputFormat

csvInputStream, csvSchema

Fields inherited from class org.apache.flink.api.common.io.FileInputFormat

currentSplit, ENUMERATE_NESTED_FILES_FLAG, enumerateNestedFiles, filePath, INFLATER_INPUT_STREAM_FACTORIES, minSplitSize, numSplits, openTimeout, READ_WHOLE_SPLIT_FLAG, splitLength, splitStart, stream, unsplittable

Method Summary

Modifier and Type	Method and Description
static RowCsvInputFormat.Builder	builder(TypeInformation<Row> typeInfo, Path... filePaths) Create a builder.
Row	nextRecord(Row record) Reads the next record from the input.
void	open(FileInputSplit split) Opens an input stream to the file defined in the input format.
boolean	reachedEnd() Method used to check if the end of the input is reached.

Methods inherited from class org.apache.flink.formats.csv.AbstractCsvInputFormat

supportsMultiPaths

Methods inherited from class org.apache.flink.api.common.io.FileInputFormat

acceptFile, close, configure, createInputSplits, decorateInputStream, extractFileExtension, getFilePath, getFilePaths, getFileStats, getFileStats, getInflaterInputStreamFactory, getInputSplitAssigner, getMinSplitSize, getNestedFileEnumeration, getNumSplits, getOpenTimeout, getSplitLength, getSplitStart, getStatistics, registerInflaterInputStreamFactory, setFilePath, setFilePath, setFilePaths, setFilePaths, setFilesFilter, setMinSplitSize, setNestedFileEnumeration, setNumSplits, setOpenTimeout, testForUnsplittable, toString

Methods inherited from class org.apache.flink.api.common.io.RichInputFormat

closeInputFormat, getRuntimeContext, openInputFormat, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Method Detail

open

public void open(FileInputSplit split)
          throws IOException

Description copied from class: FileInputFormat

Opens an input stream to the file defined in the input format. The stream is positioned at the beginning of the given split.

The stream is actually opened in an asynchronous thread to make sure any interruptions to the thread working on the input format do not reach the file system.

Specified by:

open in interface InputFormat<Row,FileInputSplit>

Overrides:

open in class AbstractCsvInputFormat<Row>

Parameters:

split - The split to be opened.

Throws:

IOException - Thrown, if the spit could not be opened due to an I/O problem.

reachedEnd

public boolean reachedEnd()

Description copied from interface: InputFormat

Method used to check if the end of the input is reached.

When this method is called, the input format it guaranteed to be opened.

Returns:

True if the end is reached, otherwise false.

nextRecord

public Row nextRecord(Row record)
               throws IOException

Description copied from interface: InputFormat

Reads the next record from the input.

When this method is called, the input format it guaranteed to be opened.

Parameters:

record - Object that may be reused.

Returns:

Read record.

Throws:

IOException - Thrown, if an I/O error occurred.

builder

public static RowCsvInputFormat.Builder builder(TypeInformation<Row> typeInfo,
                                                Path... filePaths)

Create a builder.