org.apache.flink.fs.gs.utils

Class ConfigUtils

java.lang.Object

org.apache.flink.fs.gs.utils.ConfigUtils

public class ConfigUtils
extends Object

Utilities class for configuration of Hadoop and Google Storage.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	ConfigUtils.ConfigContext Interface that provides context-specific config helper functions, factored out to support unit testing.

Constructor Summary

Constructors

Constructor and Description
ConfigUtils()

Method Summary

Modifier and Type	Method and Description
static Optional<String>	getGcsRootUrl(org.apache.hadoop.conf.Configuration hadoopConfig)
static org.apache.hadoop.conf.Configuration	getHadoopConfiguration(Configuration flinkConfig, ConfigUtils.ConfigContext configContext) Loads the Hadoop configuration, by loading from a Hadoop conf dir (if one exists) and then overlaying properties derived from the Flink config.
static Optional<com.google.auth.oauth2.GoogleCredentials>	getStorageCredentials(org.apache.hadoop.conf.Configuration hadoopConfig, ConfigUtils.ConfigContext configContext) Creates an (optional) GoogleCredentials instance for the given Hadoop config and environment.
static String	stringifyHadoopConfig(org.apache.hadoop.conf.Configuration hadoopConfig) Helper to serialize a Hadoop config to a string, for logging.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ConfigUtils

public ConfigUtils()

Method Detail

getHadoopConfiguration

public static org.apache.hadoop.conf.Configuration getHadoopConfiguration(Configuration flinkConfig,
                                                                          ConfigUtils.ConfigContext configContext)

Loads the Hadoop configuration, by loading from a Hadoop conf dir (if one exists) and then overlaying properties derived from the Flink config.

Parameters:

flinkConfig - The Flink config

configContext - The config context.

Returns:

The Hadoop config.

getStorageCredentials

public static Optional<com.google.auth.oauth2.GoogleCredentials> getStorageCredentials(org.apache.hadoop.conf.Configuration hadoopConfig,
                                                                                       ConfigUtils.ConfigContext configContext)

Creates an (optional) GoogleCredentials instance for the given Hadoop config and environment.

Parameters:

hadoopConfig - The Hadoop config.

configContext - The config context.

Returns:

The optional GoogleCredentials instance.

getGcsRootUrl

public static Optional<String> getGcsRootUrl(org.apache.hadoop.conf.Configuration hadoopConfig)

stringifyHadoopConfig

public static String stringifyHadoopConfig(org.apache.hadoop.conf.Configuration hadoopConfig)
                                    throws RuntimeException

Helper to serialize a Hadoop config to a string, for logging.

Parameters:

hadoopConfig - The Hadoop config.

Returns:

A string with the Hadoop properties.

Throws:

RuntimeException - On underlying IO failure