org.apache.flink.table.resource

Class ResourceManager

java.lang.Object

org.apache.flink.table.resource.ResourceManager

All Implemented Interfaces:

Closeable, AutoCloseable

Direct Known Subclasses:

ClientResourceManager

@Internal
public class ResourceManager
extends Object
implements Closeable

A manager for dealing with all user defined resource.

Field Summary

Fields

Modifier and Type	Field and Description
protected Path	localResourceDir
protected Map<ResourceUri,URL>	resourceInfos
protected MutableURLClassLoader	userClassLoader

Constructor Summary

Constructors

Constructor and Description
ResourceManager(ReadableConfig config, MutableURLClassLoader userClassLoader)

Method Summary

Modifier and Type	Method and Description
void	addJarConfiguration(TableConfig tableConfig) Adds the local jar resources to the given TableConfig.
protected void	checkPath(Path path, ResourceType expectedType)
void	close()
ResourceManager	copy()
static ResourceManager	createResourceManager(URL[] urls, ClassLoader parent, ReadableConfig config)
URLClassLoader	createUserClassLoader(List<ResourceUri> resourceUris)
void	declareFunctionResources(Set<ResourceUri> resourceUris) Declare a resource for function and add it to the function resource infos.
boolean	exists(Path filePath) Check whether the Path exists.
Set<URL>	getLocalJarResources() Get the local jars' URL.
Map<ResourceUri,URL>	getResources()
protected URL	getURLFromPath(Path path)
URLClassLoader	getUserClassLoader()
String	registerFileResource(ResourceUri resourceUri) Register a file resource into ResourceManager and return the absolute local file path without the scheme.
void	registerJarResources(List<ResourceUri> resourceUris) Due to anyone of the resource in list maybe fail during register, so we should stage it before actual register to guarantee transaction process.
void	registerPythonResources()
void	syncFileResource(ResourceUri resourceUri, Consumer<String> resourceGenerator) Generate a local file resource by the given resource generator and then synchronize to the path identified by the given ResourceUri.
void	unregisterFunctionResources(List<ResourceUri> resourceUris) Unregister the resource uri in function resources, when the reference count of the resource is 0, the resource will be removed from the function resources.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

localResourceDir

protected final Path localResourceDir

resourceInfos

protected final Map<ResourceUri,URL> resourceInfos

userClassLoader

protected final MutableURLClassLoader userClassLoader

Constructor Detail

ResourceManager

public ResourceManager(ReadableConfig config,
                       MutableURLClassLoader userClassLoader)

Method Detail

createResourceManager

public static ResourceManager createResourceManager(URL[] urls,
                                                    ClassLoader parent,
                                                    ReadableConfig config)

registerJarResources

public void registerJarResources(List<ResourceUri> resourceUris)
                          throws IOException

Due to anyone of the resource in list maybe fail during register, so we should stage it before actual register to guarantee transaction process. If all the resources are available, register them into the ResourceManager.

Throws:

IOException

registerFileResource

public String registerFileResource(ResourceUri resourceUri)
                            throws IOException

Register a file resource into ResourceManager and return the absolute local file path without the scheme.

If the file is remote, it will be copied to a local file, with file name suffixed with a UUID.

Parameters:

resourceUri - resource with type as ResourceType.FILE, the resource uri might or might not contain the uri scheme, or it could be a relative path.

Returns:

the absolute local file path.

Throws:

IOException

declareFunctionResources

public void declareFunctionResources(Set<ResourceUri> resourceUris)
                              throws IOException

Declare a resource for function and add it to the function resource infos. If the file is remote, it will be copied to a local file. The declared resource will not be added to resources and classloader if it is not used in the job.

Parameters:

resourceUris - the resource uri for function.

Throws:

IOException

unregisterFunctionResources

public void unregisterFunctionResources(List<ResourceUri> resourceUris)

Unregister the resource uri in function resources, when the reference count of the resource is 0, the resource will be removed from the function resources.

Parameters:

resourceUris - the uris to unregister in function resources.

registerPythonResources

public void registerPythonResources()

getUserClassLoader

public URLClassLoader getUserClassLoader()

createUserClassLoader

public URLClassLoader createUserClassLoader(List<ResourceUri> resourceUris)

getResources

public Map<ResourceUri,URL> getResources()

getLocalJarResources

public Set<URL> getLocalJarResources()

Get the local jars' URL. Return the URL corresponding to downloaded jars in the local file system for the remote jar. For the local jar, return the registered URL.

addJarConfiguration

public void addJarConfiguration(TableConfig tableConfig)

Adds the local jar resources to the given TableConfig. It implicitly considers the TableConfig.getRootConfiguration() and stores the merged result into TableConfig.getConfiguration().

copy

public ResourceManager copy()

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

exists

public boolean exists(Path filePath)
               throws IOException

Check whether the Path exists.

Throws:

IOException

syncFileResource

public void syncFileResource(ResourceUri resourceUri,
                             Consumer<String> resourceGenerator)
                      throws IOException

Generate a local file resource by the given resource generator and then synchronize to the path identified by the given ResourceUri. The path passed to resource generator should be a local path retrieved from the given ResourceUri.

NOTE: if the given ResourceUri represents a remote file path like "hdfs://path/to/file.json", then the retrieved local path will be "/localResourceDir/file-${uuid}.json"

Parameters:

resourceUri - the file resource uri to synchronize to

resourceGenerator - a consumer that generates a local copy of the file resource

Throws:

IOException

checkPath

protected void checkPath(Path path,
                         ResourceType expectedType)
                  throws IOException

Throws:

IOException

getURLFromPath

@VisibleForTesting
protected URL getURLFromPath(Path path)
                                         throws IOException

Throws:

IOException