org.apache.flink.api.common.io

Class GenericCsvInputFormat<OT>

java.lang.Object

org.apache.flink.api.common.io.RichInputFormat<OT,FileInputSplit>

org.apache.flink.api.common.io.FileInputFormat<OT>

org.apache.flink.api.common.io.DelimitedInputFormat<OT>

org.apache.flink.api.common.io.GenericCsvInputFormat<OT>

All Implemented Interfaces:

Serializable, InputFormat<OT,FileInputSplit>, InputSplitSource<FileInputSplit>

Direct Known Subclasses:

CsvInputFormat

@Internal
public abstract class GenericCsvInputFormat<OT>
extends DelimitedInputFormat<OT>

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.api.common.io.FileInputFormat

FileInputFormat.FileBaseStatistics, FileInputFormat.InputSplitOpenThread

Field Summary

Fields

Modifier and Type	Field and Description
protected int	commentCount
protected byte[]	commentPrefix
protected boolean[]	fieldIncluded
protected int	invalidLineCount
protected boolean	lineDelimiterIsLinebreak

Fields inherited from class org.apache.flink.api.common.io.DelimitedInputFormat

RECORD_DELIMITER

Fields inherited from class org.apache.flink.api.common.io.FileInputFormat

currentSplit, ENUMERATE_NESTED_FILES_FLAG, enumerateNestedFiles, filePath, INFLATER_INPUT_STREAM_FACTORIES, minSplitSize, numSplits, openTimeout, READ_WHOLE_SPLIT_FLAG, splitLength, splitStart, stream, unsplittable

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	GenericCsvInputFormat()
protected	GenericCsvInputFormat(Path filePath)

Method Summary

Modifier and Type	Method and Description
protected static void	checkAndCoSort(int[] positions, Class<?>[] types)
protected static void	checkForMonotonousOrder(int[] positions, Class<?>[] types)
void	close() Closes the input by releasing all buffers and closing the file input stream.
void	enableQuotedStringParsing(char quoteCharacter)
byte[]	getCommentPrefix()
byte[]	getFieldDelimiter()
protected FieldParser<?>[]	getFieldParsers()
protected Class<?>[]	getGenericFieldTypes()
int	getNumberOfFieldsTotal()
int	getNumberOfNonNullFields()
boolean	isLenient()
boolean	isSkippingFirstLineAsHeader()
void	open(FileInputSplit split) Opens the given input split.
protected boolean	parseRecord(Object[] holders, byte[] bytes, int offset, int numBytes)
void	setCommentPrefix(byte[] commentPrefix)
void	setCommentPrefix(char commentPrefix)
void	setCommentPrefix(String commentPrefix)
void	setCommentPrefix(String commentPrefix, Charset charset)
void	setCommentPrefix(String commentPrefix, String charsetName)
void	setFieldDelimiter(byte[] delimiter)
void	setFieldDelimiter(char delimiter)
void	setFieldDelimiter(String delimiter)
protected void	setFieldsGeneric(boolean[] includedMask, Class<?>[] fieldTypes)
protected void	setFieldsGeneric(int[] sourceFieldIndices, Class<?>[] fieldTypes)
protected void	setFieldTypesGeneric(Class<?>... fieldTypes)
void	setLenient(boolean lenient)
void	setSkipFirstLineAsHeader(boolean skipFirstLine)
protected int	skipFields(byte[] bytes, int startPos, int limit, byte[] delim)

Methods inherited from class org.apache.flink.api.common.io.DelimitedInputFormat

configure, getBufferSize, getDelimiter, getLineLengthLimit, getNumLineSamples, getStatistics, loadGlobalConfigParams, nextRecord, reachedEnd, readLine, readRecord, setBufferSize, setDelimiter, setDelimiter, setDelimiter, setLineLengthLimit, setNumLineSamples

Methods inherited from class org.apache.flink.api.common.io.FileInputFormat

acceptFile, createInputSplits, decorateInputStream, extractFileExtension, getFilePath, getFileStats, getInflaterInputStreamFactory, getInputSplitAssigner, getMinSplitSize, getNumSplits, getOpenTimeout, getSplitLength, getSplitStart, registerInflaterInputStreamFactory, setFilePath, setFilePath, setMinSplitSize, setNumSplits, setOpenTimeout, testForUnsplittable, toString

Methods inherited from class org.apache.flink.api.common.io.RichInputFormat

getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

lineDelimiterIsLinebreak

protected boolean lineDelimiterIsLinebreak

commentCount

protected transient int commentCount

invalidLineCount

protected transient int invalidLineCount

fieldIncluded

protected boolean[] fieldIncluded

commentPrefix

protected byte[] commentPrefix

Constructor Detail

GenericCsvInputFormat

protected GenericCsvInputFormat()

GenericCsvInputFormat

protected GenericCsvInputFormat(Path filePath)

Method Detail

getNumberOfFieldsTotal

public int getNumberOfFieldsTotal()

getNumberOfNonNullFields

public int getNumberOfNonNullFields()

getCommentPrefix

public byte[] getCommentPrefix()

setCommentPrefix

public void setCommentPrefix(byte[] commentPrefix)

setCommentPrefix

public void setCommentPrefix(char commentPrefix)

setCommentPrefix

public void setCommentPrefix(String commentPrefix)

setCommentPrefix

public void setCommentPrefix(String commentPrefix,
                             String charsetName)
                      throws IllegalCharsetNameException,
                             UnsupportedCharsetException

Throws:

IllegalCharsetNameException

UnsupportedCharsetException

setCommentPrefix

public void setCommentPrefix(String commentPrefix,
                             Charset charset)

getFieldDelimiter

public byte[] getFieldDelimiter()

setFieldDelimiter

public void setFieldDelimiter(byte[] delimiter)

setFieldDelimiter

public void setFieldDelimiter(char delimiter)

setFieldDelimiter

public void setFieldDelimiter(String delimiter)

isLenient

public boolean isLenient()

setLenient

public void setLenient(boolean lenient)

isSkippingFirstLineAsHeader

public boolean isSkippingFirstLineAsHeader()

setSkipFirstLineAsHeader

public void setSkipFirstLineAsHeader(boolean skipFirstLine)

enableQuotedStringParsing

public void enableQuotedStringParsing(char quoteCharacter)

getFieldParsers

protected FieldParser<?>[] getFieldParsers()

getGenericFieldTypes

protected Class<?>[] getGenericFieldTypes()

setFieldTypesGeneric

protected void setFieldTypesGeneric(Class<?>... fieldTypes)

setFieldsGeneric

protected void setFieldsGeneric(int[] sourceFieldIndices,
                                Class<?>[] fieldTypes)

setFieldsGeneric

protected void setFieldsGeneric(boolean[] includedMask,
                                Class<?>[] fieldTypes)

open

public void open(FileInputSplit split)
          throws IOException

Description copied from class: DelimitedInputFormat

Opens the given input split. This method opens the input stream to the specified file, allocates read buffers and positions the stream at the correct position, making sure that any partial record at the beginning is skipped.

Specified by:

open in interface InputFormat<OT,FileInputSplit>

Overrides:

open in class DelimitedInputFormat<OT>

Parameters:

split - The input split to open.

Throws:

IOException - Thrown, if the spit could not be opened due to an I/O problem.

See Also:

FileInputFormat.open(org.apache.flink.core.fs.FileInputSplit)

close

public void close()
           throws IOException

Description copied from class: DelimitedInputFormat

Closes the input by releasing all buffers and closing the file input stream.

Specified by:

close in interface InputFormat<OT,FileInputSplit>

Overrides:

close in class DelimitedInputFormat<OT>

Throws:

IOException - Thrown, if the closing of the file stream causes an I/O error.

parseRecord

protected boolean parseRecord(Object[] holders,
                              byte[] bytes,
                              int offset,
                              int numBytes)
                       throws ParseException

Throws:

ParseException

skipFields

protected int skipFields(byte[] bytes,
                         int startPos,
                         int limit,
                         byte[] delim)

checkAndCoSort

protected static void checkAndCoSort(int[] positions,
                                     Class<?>[] types)

checkForMonotonousOrder

protected static void checkForMonotonousOrder(int[] positions,
                                              Class<?>[] types)