org.apache.flink.api.java.io

Class CsvInputFormat<OUT>

java.lang.Object

org.apache.flink.api.common.io.RichInputFormat<OT,FileInputSplit>

org.apache.flink.api.common.io.FileInputFormat<OT>

org.apache.flink.api.common.io.DelimitedInputFormat<OT>

org.apache.flink.api.common.io.GenericCsvInputFormat<OUT>

org.apache.flink.api.java.io.CsvInputFormat<OUT>

Type Parameters:

OUT -

All Implemented Interfaces:

Serializable, CheckpointableInputFormat<FileInputSplit,Long>, InputFormat<OUT,FileInputSplit>, InputSplitSource<FileInputSplit>

Direct Known Subclasses:

PojoCsvInputFormat, RowCsvInputFormat, TupleCsvInputFormat

@Internal
public abstract class CsvInputFormat<OUT>
extends GenericCsvInputFormat<OUT>

InputFormat that reads csv files.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.api.common.io.FileInputFormat

FileInputFormat.FileBaseStatistics, FileInputFormat.InputSplitOpenThread

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEFAULT_FIELD_DELIMITER
static String	DEFAULT_LINE_DELIMITER
protected Object[]	parsedValues

Fields inherited from class org.apache.flink.api.common.io.GenericCsvInputFormat

commentCount, commentPrefix, fieldIncluded, invalidLineCount, lineDelimiterIsLinebreak

Fields inherited from class org.apache.flink.api.common.io.DelimitedInputFormat

RECORD_DELIMITER

Fields inherited from class org.apache.flink.api.common.io.FileInputFormat

currentSplit, ENUMERATE_NESTED_FILES_FLAG, enumerateNestedFiles, filePath, INFLATER_INPUT_STREAM_FACTORIES, minSplitSize, numSplits, openTimeout, READ_WHOLE_SPLIT_FLAG, splitLength, splitStart, stream, unsplittable

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	CsvInputFormat(Path filePath)

Method Summary

Modifier and Type	Method and Description
protected static boolean[]	createDefaultMask(int size)
protected abstract OUT	fillRecord(OUT reuse, Object[] parsedValues)
Class<?>[]	getFieldTypes()
OUT	nextRecord(OUT record) Reads the next record from the input.
void	open(FileInputSplit split) Opens the given input split.
OUT	readRecord(OUT reuse, byte[] bytes, int offset, int numBytes) This function parses the given byte array which represents a serialized record.
protected static boolean[]	toBooleanMask(int[] sourceFieldIndices)
String	toString()

Methods inherited from class org.apache.flink.api.common.io.GenericCsvInputFormat

checkAndCoSort, checkForMonotonousOrder, close, enableQuotedStringParsing, getCommentPrefix, getFieldDelimiter, getFieldParsers, getGenericFieldTypes, getNumberOfFieldsTotal, getNumberOfNonNullFields, isLenient, isSkippingFirstLineAsHeader, parseRecord, setCharset, setCommentPrefix, setFieldDelimiter, setFieldsGeneric, setFieldsGeneric, setFieldTypesGeneric, setLenient, setSkipFirstLineAsHeader, skipFields, supportsMultiPaths

Methods inherited from class org.apache.flink.api.common.io.DelimitedInputFormat

configure, getBufferSize, getCharset, getCurrentState, getDelimiter, getLineLengthLimit, getNumLineSamples, getStatistics, loadConfigParameters, loadGlobalConfigParams, reachedEnd, readLine, reopen, setBufferSize, setDelimiter, setDelimiter, setDelimiter, setLineLengthLimit, setNumLineSamples

Methods inherited from class org.apache.flink.api.common.io.FileInputFormat

acceptFile, createInputSplits, decorateInputStream, extractFileExtension, getFilePath, getFilePaths, getFileStats, getFileStats, getInflaterInputStreamFactory, getInputSplitAssigner, getMinSplitSize, getNestedFileEnumeration, getNumSplits, getOpenTimeout, getSplitLength, getSplitStart, registerInflaterInputStreamFactory, setFilePath, setFilePath, setFilePaths, setFilePaths, setFilesFilter, setMinSplitSize, setNestedFileEnumeration, setNumSplits, setOpenTimeout, testForUnsplittable

Methods inherited from class org.apache.flink.api.common.io.RichInputFormat

closeInputFormat, getRuntimeContext, openInputFormat, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_LINE_DELIMITER

public static final String DEFAULT_LINE_DELIMITER

See Also:

Constant Field Values

DEFAULT_FIELD_DELIMITER

public static final String DEFAULT_FIELD_DELIMITER

See Also:

Constant Field Values

parsedValues

protected transient Object[] parsedValues

Constructor Detail

CsvInputFormat

protected CsvInputFormat(Path filePath)

Method Detail

open

public void open(FileInputSplit split)
          throws IOException

Description copied from class: DelimitedInputFormat

Opens the given input split. This method opens the input stream to the specified file, allocates read buffers and positions the stream at the correct position, making sure that any partial record at the beginning is skipped.

Specified by:

open in interface InputFormat<OUT,FileInputSplit>

Overrides:

open in class GenericCsvInputFormat<OUT>

Parameters:

split - The input split to open.

Throws:

IOException - Thrown, if the spit could not be opened due to an I/O problem.

See Also:

FileInputFormat.open(org.apache.flink.core.fs.FileInputSplit)

nextRecord

public OUT nextRecord(OUT record)
               throws IOException

Description copied from interface: InputFormat

Reads the next record from the input.

When this method is called, the input format it guaranteed to be opened.

Specified by:

nextRecord in interface InputFormat<OUT,FileInputSplit>

Overrides:

nextRecord in class DelimitedInputFormat<OUT>

Parameters:

record - Object that may be reused.

Returns:

Read record.

Throws:

IOException - Thrown, if an I/O error occurred.

readRecord

public OUT readRecord(OUT reuse,
                      byte[] bytes,
                      int offset,
                      int numBytes)
               throws IOException

Description copied from class: DelimitedInputFormat

This function parses the given byte array which represents a serialized record. The function returns a valid record or throws an IOException.

Specified by:

readRecord in class DelimitedInputFormat<OUT>

Parameters:

reuse - An optionally reusable object.

bytes - Binary data of serialized records.

offset - The offset where to start to read the record data.

numBytes - The number of bytes that can be read starting at the offset position.

Returns:

Returns the read record if it was successfully deserialized.

Throws:

IOException - if the record could not be read.

fillRecord

protected abstract OUT fillRecord(OUT reuse,
                                  Object[] parsedValues)

getFieldTypes

public Class<?>[] getFieldTypes()

createDefaultMask

protected static boolean[] createDefaultMask(int size)

toBooleanMask

protected static boolean[] toBooleanMask(int[] sourceFieldIndices)

toString

public String toString()

Overrides:

toString in class FileInputFormat<OUT>