@Internal public class PartitionLoader extends Object implements Closeable
This provide two interface to load: 1.loadPartition(java.util.LinkedHashMap<java.lang.String, java.lang.String>, java.util.List<org.apache.flink.core.fs.Path>)
: load temporary partitioned
files, if it is new partition, will create partition to meta store. 2.loadNonPartition(java.util.List<org.apache.flink.core.fs.Path>)
:
just rename all files to final output path.
TODO: src and dest may be on different FS.
Constructor and Description |
---|
PartitionLoader(boolean overwrite,
FileSystem sourceFs,
TableMetaStoreFactory factory,
boolean isToLocal,
ObjectIdentifier identifier,
List<PartitionCommitPolicy> policies) |
Modifier and Type | Method and Description |
---|---|
void |
close() |
void |
loadEmptyPartition(LinkedHashMap<String,String> partSpec)
The flink job does not write data to the partition, but the corresponding partition needs to
be created or updated.
|
void |
loadNonPartition(List<Path> srcDirs)
Load a non-partition files to output path.
|
void |
loadPartition(LinkedHashMap<String,String> partSpec,
List<Path> srcDirs)
Load a single partition.
|
public PartitionLoader(boolean overwrite, FileSystem sourceFs, TableMetaStoreFactory factory, boolean isToLocal, ObjectIdentifier identifier, List<PartitionCommitPolicy> policies) throws Exception
Exception
public void loadPartition(LinkedHashMap<String,String> partSpec, List<Path> srcDirs) throws Exception
Exception
public void loadNonPartition(List<Path> srcDirs) throws Exception
Exception
public void loadEmptyPartition(LinkedHashMap<String,String> partSpec) throws Exception
The partition does not exist, create it.
The partition exists:
if overwrite is true, delete the path, then create it; if overwrite is false, do nothing;
Exception
public void close() throws IOException
close
in interface Closeable
close
in interface AutoCloseable
IOException
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.