org.apache.flink.connector.file.table

Class PartitionLoader

java.lang.Object

org.apache.flink.connector.file.table.PartitionLoader

All Implemented Interfaces:

Closeable, AutoCloseable

@Internal
public class PartitionLoader
extends Object
implements Closeable

Loader to temporary files to final output path and meta store. According to overwrite, the loader will delete the previous data.

This provide two interface to load: 1.loadPartition(java.util.LinkedHashMap<java.lang.String, java.lang.String>, java.util.List<org.apache.flink.core.fs.Path>, boolean): load temporary partitioned files, if it is new partition, will create partition to meta store. 2.loadNonPartition(java.util.List<org.apache.flink.core.fs.Path>, boolean): just rename all files to final output path.

TODO: src and dest may be on different FS.

Constructor Summary

Constructors

Constructor and Description
PartitionLoader(boolean overwrite, FileSystem sourceFs, TableMetaStoreFactory factory, boolean isToLocal, ObjectIdentifier identifier, List<PartitionCommitPolicy> policies)

Method Summary

Modifier and Type	Method and Description
void	close()
void	loadEmptyPartition(LinkedHashMap<String,String> partSpec) The flink job does not write data to the partition, but the corresponding partition needs to be created or updated.
void	loadNonPartition(List<Path> srcPaths, boolean srcPathIsDir) Load a non-partition files to output path.
void	loadPartition(LinkedHashMap<String,String> partSpec, List<Path> srcPaths, boolean srcPathIsDir) Load a single partition.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PartitionLoader

public PartitionLoader(boolean overwrite,
                       FileSystem sourceFs,
                       TableMetaStoreFactory factory,
                       boolean isToLocal,
                       ObjectIdentifier identifier,
                       List<PartitionCommitPolicy> policies)
                throws Exception

Throws:

Exception

Method Detail

loadPartition

public void loadPartition(LinkedHashMap<String,String> partSpec,
                          List<Path> srcPaths,
                          boolean srcPathIsDir)
                   throws Exception

Load a single partition.

Parameters:

partSpec - the specification for the single partition

srcPaths - the paths for the files used to load to the single partition

srcPathIsDir - whether the every path in is directory or not. If true, it will load the files under the directory of the every path. If false, every path in is considered as single file, and it will load the single file for every path.

Throws:

Exception

loadNonPartition

public void loadNonPartition(List<Path> srcPaths,
                             boolean srcPathIsDir)
                      throws Exception

Load a non-partition files to output path.

Parameters:

srcPaths - the paths for the files used to load to the single partition

srcPathIsDir - whether the every path in is directory or not. If true, it will load the files under the directory of the every path. If false, every path in is considered as single file, and it will load the single file for every path.

Throws:

Exception

loadEmptyPartition

public void loadEmptyPartition(LinkedHashMap<String,String> partSpec)
                        throws Exception

The flink job does not write data to the partition, but the corresponding partition needs to be created or updated.

The partition does not exist, create it.

The partition exists:

      if overwrite is true, delete the path, then create it;
      if overwrite is false, do nothing;
 

Throws:

Exception

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException