org.apache.flink.connector.file.src

Class FileSourceSplit

java.lang.Object

org.apache.flink.connector.file.src.FileSourceSplit

All Implemented Interfaces:

Serializable, SourceSplit

Direct Known Subclasses:

HiveSourceSplit

@PublicEvolving
public class FileSourceSplit
extends Object
implements SourceSplit, Serializable

A SourceSplit that represents a file, or a region of a file.

The split has an offset and an end, which defines the region of the file represented by the split. For splits representing the while file, the offset is zero and the length is the file size.

The split may furthermore have a "reader position", which is the checkpointed position from a reader previously reading this split. This position is typically null when the split is assigned from the enumerator to the readers, and is non-null when the readers checkpoint their state in a file source split.

This class is Serializable for convenience. For Flink's internal serialization (both for RPC and for checkpoints), the FileSourceSplitSerializer is used.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
FileSourceSplit(String id, Path filePath, long offset, long length) Deprecated. You should use FileSourceSplit(String, Path, long, long, long, long)
FileSourceSplit(String id, Path filePath, long offset, long length, long fileModificationTime, long fileSize) Constructs a split with host information.
FileSourceSplit(String id, Path filePath, long offset, long length, long fileModificationTime, long fileSize, String... hostnames) Constructs a split with host information.
FileSourceSplit(String id, Path filePath, long offset, long length, long fileModificationTime, long fileSize, String[] hostnames, CheckpointedPosition readerPosition) Constructs a split with host information.
FileSourceSplit(String id, Path filePath, long offset, long length, String... hostnames) Deprecated. You should use FileSourceSplit(String, Path, long, long, long, long, String...)
FileSourceSplit(String id, Path filePath, long offset, long length, String[] hostnames, CheckpointedPosition readerPosition) Deprecated. You should use FileSourceSplit(String, Path, long, long, long, long, String[], CheckpointedPosition)

Method Summary

Modifier and Type	Method and Description
long	fileModificationTime() Returns the modification time of the file, from FileStatus.getModificationTime().
long	fileSize() Returns the full file size in bytes, from FileStatus.getLen().
Optional<CheckpointedPosition>	getReaderPosition() Gets the (checkpointed) position of the reader, if set.
String[]	hostnames() Gets the hostnames of the nodes storing the file range described by this split.
long	length() Returns the number of bytes in the file region described by this source split.
long	offset() Returns the start of the file region referenced by this source split.
Path	path() Gets the file's path.
String	splitId() Get the split id of this source split.
String	toString()
FileSourceSplit	updateWithCheckpointedPosition(CheckpointedPosition position) Creates a copy of this split where the checkpointed position is replaced by the given new position.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

FileSourceSplit

public FileSourceSplit(String id,
                       Path filePath,
                       long offset,
                       long length,
                       long fileModificationTime,
                       long fileSize)

Constructs a split with host information.

Parameters:

id - The unique ID of this source split.

filePath - The path to the file.

offset - The start (inclusive) of the split's rage in the file.

length - The number of bytes in the split (starting from the offset)

fileModificationTime - The modification time of the file

fileSize - The size of the full file

FileSourceSplit

public FileSourceSplit(String id,
                       Path filePath,
                       long offset,
                       long length,
                       long fileModificationTime,
                       long fileSize,
                       String... hostnames)

Constructs a split with host information.

Parameters:

filePath - The path to the file.

offset - The start (inclusive) of the split's rage in the file.

length - The number of bytes in the split (starting from the offset)

fileModificationTime - The modification time of the file

fileSize - The size of the full file

hostnames - The hostnames of the nodes storing the split's file range.

FileSourceSplit

public FileSourceSplit(String id,
                       Path filePath,
                       long offset,
                       long length,
                       long fileModificationTime,
                       long fileSize,
                       String[] hostnames,
                       @Nullable
                       CheckpointedPosition readerPosition)

Constructs a split with host information.

Parameters:

filePath - The path to the file.

offset - The start (inclusive) of the split's rage in the file.

length - The number of bytes in the split (starting from the offset)

fileModificationTime - The modification time of the file

fileSize - The size of the full file

hostnames - The hostnames of the nodes storing the split's file range.

FileSourceSplit

@Deprecated
public FileSourceSplit(String id,
                                   Path filePath,
                                   long offset,
                                   long length)

Deprecated. You should use FileSourceSplit(String, Path, long, long, long, long)

FileSourceSplit

@Deprecated
public FileSourceSplit(String id,
                                   Path filePath,
                                   long offset,
                                   long length,
                                   String... hostnames)

Deprecated. You should use FileSourceSplit(String, Path, long, long, long, long, String...)

FileSourceSplit

@Deprecated
public FileSourceSplit(String id,
                                   Path filePath,
                                   long offset,
                                   long length,
                                   String[] hostnames,
                                   @Nullable
                                   CheckpointedPosition readerPosition)

Deprecated. You should use FileSourceSplit(String, Path, long, long, long, long, String[], CheckpointedPosition)

Method Detail

splitId

public String splitId()

Description copied from interface: SourceSplit

Get the split id of this source split.

Specified by:

splitId in interface SourceSplit

Returns:

id of this source split.

path

public Path path()

Gets the file's path.

offset

public long offset()

Returns the start of the file region referenced by this source split. The position is inclusive, the value indicates the first byte that is part of the split.

length

public long length()

Returns the number of bytes in the file region described by this source split.

fileModificationTime

public long fileModificationTime()

Returns the modification time of the file, from FileStatus.getModificationTime().

fileSize

public long fileSize()

Returns the full file size in bytes, from FileStatus.getLen().

hostnames

public String[] hostnames()

Gets the hostnames of the nodes storing the file range described by this split. The returned array is empty, if no host information is available.

Host information is typically only available on specific file systems, like HDFS.

getReaderPosition

public Optional<CheckpointedPosition> getReaderPosition()

Gets the (checkpointed) position of the reader, if set. This value is typically absent for splits when assigned from the enumerator to the readers, and present when the splits are recovered from a checkpoint.

updateWithCheckpointedPosition

public FileSourceSplit updateWithCheckpointedPosition(@Nullable
                                                      CheckpointedPosition position)

Creates a copy of this split where the checkpointed position is replaced by the given new position.

IMPORTANT: Subclasses that add additional information to the split must override this method to return that subclass type. This contract is enforced by checks in the file source implementation. We did not try to enforce this contract via generics in this split class, because it leads to very ugly and verbose use of generics.

toString

public String toString()

Overrides:

toString in class Object