org.apache.flink.connector.file.table

Class PartitionTempFileManager

java.lang.Object

org.apache.flink.connector.file.table.PartitionTempFileManager

@Internal
public class PartitionTempFileManager
extends Object

Manage temporary files for writing files. Use special rules to organize directories for temporary files.

Temporary file directory contains the following directory parts: 1.temporary base path directory. 2.task id directory. 3.directories to specify partitioning. 4.data files. eg: /tmp/task-0-attempt-0/p0=1/p1=2/fileName.

Constructor Summary

Constructors

Constructor and Description
PartitionTempFileManager(FileSystemFactory factory, Path tmpPath, int taskNumber, int attemptNumber)
PartitionTempFileManager(FileSystemFactory factory, Path tmpPath, int taskNumber, int attemptNumber, OutputFileConfig outputFileConfig)

Method Summary

Modifier and Type	Method and Description
static Map<LinkedHashMap<String,String>,List<Path>>	collectPartSpecToPaths(FileSystem fs, List<Path> taskPaths, int partColSize) Collect all partitioned paths, aggregate according to partition spec.
Path	createPartitionDir(String... partitions) Generate a new partition directory with partitions.
static List<Path>	listTaskTemporaryPaths(FileSystem fs, Path basePath, java.util.function.BiPredicate<Integer,Integer> taskAttemptFilter) Returns task temporary paths in this checkpoint.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PartitionTempFileManager

public PartitionTempFileManager(FileSystemFactory factory,
                                Path tmpPath,
                                int taskNumber,
                                int attemptNumber)
                         throws IOException

Throws:

IOException

PartitionTempFileManager

public PartitionTempFileManager(FileSystemFactory factory,
                                Path tmpPath,
                                int taskNumber,
                                int attemptNumber,
                                OutputFileConfig outputFileConfig)
                         throws IOException

Throws:

IOException

Method Detail

createPartitionDir

public Path createPartitionDir(String... partitions)

Generate a new partition directory with partitions.

listTaskTemporaryPaths

public static List<Path> listTaskTemporaryPaths(FileSystem fs,
                                                Path basePath,
                                                java.util.function.BiPredicate<Integer,Integer> taskAttemptFilter)
                                         throws Exception

Returns task temporary paths in this checkpoint.

Throws:

Exception

collectPartSpecToPaths

public static Map<LinkedHashMap<String,String>,List<Path>> collectPartSpecToPaths(FileSystem fs,
                                                                                  List<Path> taskPaths,
                                                                                  int partColSize)

Collect all partitioned paths, aggregate according to partition spec.