public class FileStoreCommitImpl extends Object implements FileStoreCommit
FileStoreCommit
.
This class provides an atomic commit method to the user.
commit(org.apache.flink.table.store.file.manifest.ManifestCommittable, java.util.Map<java.lang.String, java.lang.String>)
, if user cannot determine if this commit
is done before, user should first call filterCommitted(java.util.List<org.apache.flink.table.store.file.manifest.ManifestCommittable>)
.
lock
(if provided) or the atomic
rename of the file system to ensure atomicity.
NOTE: If you want to modify this class, any exception during commit MUST NOT BE IGNORED. They must be thrown to restart the job. It is recommended to run FileStoreCommitTest thousands of times to make sure that your changes is correct.
Constructor and Description |
---|
FileStoreCommitImpl(long schemaId,
String commitUser,
org.apache.flink.table.types.logical.RowType partitionType,
FileStorePathFactory pathFactory,
SnapshotManager snapshotManager,
ManifestFile.Factory manifestFileFactory,
ManifestList.Factory manifestListFactory,
FileStoreScan scan,
int numBucket,
org.apache.flink.configuration.MemorySize manifestTargetSize,
int manifestMergeMinCount,
Comparator<org.apache.flink.table.data.RowData> keyComparator) |
Modifier and Type | Method and Description |
---|---|
void |
commit(ManifestCommittable committable,
Map<String,String> properties)
Commit from manifest committable.
|
List<ManifestCommittable> |
filterCommitted(List<ManifestCommittable> committableList)
Find out which manifest committable need to be retried when recovering from the failure.
|
void |
overwrite(List<Map<String,String>> partitions,
ManifestCommittable committable,
Map<String,String> properties)
Overwrite multiple partitions from manifest committable.
|
FileStoreCommit |
withCreateEmptyCommit(boolean createEmptyCommit) |
FileStoreCommit |
withLock(Lock lock)
With global lock.
|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
overwrite
public FileStoreCommitImpl(long schemaId, String commitUser, org.apache.flink.table.types.logical.RowType partitionType, FileStorePathFactory pathFactory, SnapshotManager snapshotManager, ManifestFile.Factory manifestFileFactory, ManifestList.Factory manifestListFactory, FileStoreScan scan, int numBucket, org.apache.flink.configuration.MemorySize manifestTargetSize, int manifestMergeMinCount, @Nullable Comparator<org.apache.flink.table.data.RowData> keyComparator)
public FileStoreCommit withLock(Lock lock)
FileStoreCommit
withLock
in interface FileStoreCommit
public FileStoreCommit withCreateEmptyCommit(boolean createEmptyCommit)
withCreateEmptyCommit
in interface FileStoreCommit
public List<ManifestCommittable> filterCommitted(List<ManifestCommittable> committableList)
FileStoreCommit
filterCommitted
in interface FileStoreCommit
public void commit(ManifestCommittable committable, Map<String,String> properties)
FileStoreCommit
commit
in interface FileStoreCommit
public void overwrite(List<Map<String,String>> partitions, ManifestCommittable committable, Map<String,String> properties)
FileStoreCommit
overwrite
in interface FileStoreCommit
partitions
- A list of partition Map
s that maps each partition key to a
partition value. Depending on the user-defined statement, the partition might not include
all partition keys. Also note that this partition does not necessarily equal to the
partitions of the newly added key-values. This is just the partition to be cleaned up.Copyright © 2019–2023 The Apache Software Foundation. All rights reserved.