org.apache.flink.runtime.clusterframework

Class BootstrapTools

java.lang.Object

org.apache.flink.runtime.clusterframework.BootstrapTools

public class BootstrapTools
extends Object

Tools for starting JobManager and TaskManager processes, including the Actor Systems used to run the JobManager and TaskManager actors.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	BootstrapTools.ActorSystemExecutorConfiguration Configuration interface for ActorSystem underlying executor.
static class	BootstrapTools.FixedThreadPoolExecutorConfiguration Configuration for a fixed thread pool executor.
static class	BootstrapTools.ForkJoinExecutorConfiguration Configuration for a fork join executor.

Method Summary

Modifier and Type	Method and Description
static int	calculateHeapSize(int memory, Configuration conf) Calculate heap size after cut-off.
static Configuration	cloneConfiguration(Configuration configuration) Clones the given configuration and resets instance specific config options.
static String	escapeForDifferentOS(String value) Escape all the dynamic property values.
static String	escapeWithDoubleQuote(String value)
static String	escapeWithSingleQuote(String value)
static String	getDynamicPropertiesAsString(Configuration baseConfig, Configuration targetConfig) Get dynamic properties based on two Flink configurations.
static Map<String,String>	getEnvironmentVariables(String envPrefix, Configuration flinkConfiguration) Method to extract environment variables from the flinkConfiguration based on the given prefix String.
static String	getStartCommand(String template, Map<String,String> startCommandValues) Replaces placeholders in the template start command with values from startCommandValues.
static String	getTaskManagerShellCommand(Configuration flinkConfig, ContaineredTaskManagerParameters tmParams, String configDirectory, String logDirectory, boolean hasLogback, boolean hasLog4j, boolean hasKrb5, Class<?> mainClass, String mainArgs) Generates the shell command to start a task manager.
static org.apache.commons.cli.Option	newDynamicPropertiesOption() Get an instance of the dynamic properties option.
static Configuration	parseDynamicProperties(org.apache.commons.cli.CommandLine cmd) Parse the dynamic properties (passed on the command line).
static akka.actor.ActorSystem	startActorSystem(Configuration configuration, String listeningAddress, int listeningPort, org.slf4j.Logger logger) Starts an Actor System at a specific port.
static akka.actor.ActorSystem	startActorSystem(Configuration configuration, String listeningAddress, int listeningPort, org.slf4j.Logger logger, BootstrapTools.ActorSystemExecutorConfiguration actorSystemExecutorConfiguration) Starts an Actor System at a specific port.
static akka.actor.ActorSystem	startActorSystem(Configuration configuration, String actorSystemName, String listeningAddress, int listeningPort, org.slf4j.Logger logger, BootstrapTools.ActorSystemExecutorConfiguration actorSystemExecutorConfiguration) Starts an Actor System at a specific port.
static akka.actor.ActorSystem	startActorSystem(Configuration configuration, String listeningAddress, String portRangeDefinition, org.slf4j.Logger logger) Starts an ActorSystem with the given configuration listening at the address/ports.
static akka.actor.ActorSystem	startActorSystem(Configuration configuration, String listeningAddress, String portRangeDefinition, org.slf4j.Logger logger, BootstrapTools.ActorSystemExecutorConfiguration actorSystemExecutorConfiguration) Starts an ActorSystem with the given configuration listening at the address/ports.
static akka.actor.ActorSystem	startActorSystem(Configuration configuration, String actorSystemName, String listeningAddress, String portRangeDefinition, org.slf4j.Logger logger, BootstrapTools.ActorSystemExecutorConfiguration actorSystemExecutorConfiguration) Starts an ActorSystem with the given configuration listening at the address/ports.
static void	substituteDeprecatedConfigKey(Configuration config, String deprecated, String designated) Sets the value of a new config key to the value of a deprecated config key.
static void	substituteDeprecatedConfigPrefix(Configuration config, String deprecatedPrefix, String designatedPrefix) Sets the value of a new config key to the value of a deprecated config key.
static void	updateTmpDirectoriesInConfiguration(Configuration configuration, String defaultDirs) Set temporary configuration directories if necessary.
static void	writeConfiguration(Configuration cfg, File file) Writes a Flink YAML config file from a Flink Configuration object.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

startActorSystem

public static akka.actor.ActorSystem startActorSystem(Configuration configuration,
                                                      String listeningAddress,
                                                      String portRangeDefinition,
                                                      org.slf4j.Logger logger)
                                               throws Exception

Starts an ActorSystem with the given configuration listening at the address/ports.

Parameters:

configuration - The Flink configuration

listeningAddress - The address to listen at.

portRangeDefinition - The port range to choose a port from.

logger - The logger to output log information.

Returns:

The ActorSystem which has been started

Throws:

Exception - Thrown when actor system cannot be started in specified port range

startActorSystem

public static akka.actor.ActorSystem startActorSystem(Configuration configuration,
                                                      String listeningAddress,
                                                      String portRangeDefinition,
                                                      org.slf4j.Logger logger,
                                                      @Nonnull
                                                      BootstrapTools.ActorSystemExecutorConfiguration actorSystemExecutorConfiguration)
                                               throws Exception

Starts an ActorSystem with the given configuration listening at the address/ports.

Parameters:

configuration - The Flink configuration

listeningAddress - The address to listen at.

portRangeDefinition - The port range to choose a port from.

logger - The logger to output log information.

actorSystemExecutorConfiguration - configuration for the ActorSystem's underlying executor

Returns:

The ActorSystem which has been started

Throws:

Exception - Thrown when actor system cannot be started in specified port range

startActorSystem

public static akka.actor.ActorSystem startActorSystem(Configuration configuration,
                                                      String actorSystemName,
                                                      String listeningAddress,
                                                      String portRangeDefinition,
                                                      org.slf4j.Logger logger,
                                                      @Nonnull
                                                      BootstrapTools.ActorSystemExecutorConfiguration actorSystemExecutorConfiguration)
                                               throws Exception

Starts an ActorSystem with the given configuration listening at the address/ports.

Parameters:

configuration - The Flink configuration

actorSystemName - Name of the started ActorSystem

listeningAddress - The address to listen at.

portRangeDefinition - The port range to choose a port from.

logger - The logger to output log information.

actorSystemExecutorConfiguration - configuration for the ActorSystem's underlying executor

Returns:

The ActorSystem which has been started

Throws:

Exception - Thrown when actor system cannot be started in specified port range

startActorSystem

public static akka.actor.ActorSystem startActorSystem(Configuration configuration,
                                                      String listeningAddress,
                                                      int listeningPort,
                                                      org.slf4j.Logger logger)
                                               throws Exception

Starts an Actor System at a specific port.

Parameters:

configuration - The Flink configuration.

listeningAddress - The address to listen at.

listeningPort - The port to listen at.

logger - the logger to output log information.

Returns:

The ActorSystem which has been started.

Throws:

Exception

startActorSystem

public static akka.actor.ActorSystem startActorSystem(Configuration configuration,
                                                      String listeningAddress,
                                                      int listeningPort,
                                                      org.slf4j.Logger logger,
                                                      BootstrapTools.ActorSystemExecutorConfiguration actorSystemExecutorConfiguration)
                                               throws Exception

Starts an Actor System at a specific port.

Parameters:

configuration - The Flink configuration.

listeningAddress - The address to listen at.

listeningPort - The port to listen at.

logger - the logger to output log information.

actorSystemExecutorConfiguration - configuration for the ActorSystem's underlying executor

Returns:

The ActorSystem which has been started.

Throws:

Exception

startActorSystem

public static akka.actor.ActorSystem startActorSystem(Configuration configuration,
                                                      String actorSystemName,
                                                      String listeningAddress,
                                                      int listeningPort,
                                                      org.slf4j.Logger logger,
                                                      BootstrapTools.ActorSystemExecutorConfiguration actorSystemExecutorConfiguration)
                                               throws Exception

Starts an Actor System at a specific port.

Parameters:

configuration - The Flink configuration.

actorSystemName - Name of the started ActorSystem

listeningAddress - The address to listen at.

listeningPort - The port to listen at.

logger - the logger to output log information.

actorSystemExecutorConfiguration - configuration for the ActorSystem's underlying executor

Returns:

The ActorSystem which has been started.

Throws:

Exception

writeConfiguration

public static void writeConfiguration(Configuration cfg,
                                      File file)
                               throws IOException

Writes a Flink YAML config file from a Flink Configuration object.

Parameters:

cfg - The Flink config

file - The File to write to

Throws:

IOException

substituteDeprecatedConfigKey

public static void substituteDeprecatedConfigKey(Configuration config,
                                                 String deprecated,
                                                 String designated)

Sets the value of a new config key to the value of a deprecated config key.

Parameters:

config - Config to write

deprecated - The old config key

designated - The new config key

substituteDeprecatedConfigPrefix

public static void substituteDeprecatedConfigPrefix(Configuration config,
                                                    String deprecatedPrefix,
                                                    String designatedPrefix)

Sets the value of a new config key to the value of a deprecated config key. Taking into account the changed prefix.

Parameters:

config - Config to write

deprecatedPrefix - Old prefix of key

designatedPrefix - New prefix of key

newDynamicPropertiesOption

public static org.apache.commons.cli.Option newDynamicPropertiesOption()

Get an instance of the dynamic properties option.

Dynamic properties allow the user to specify additional configuration values with -D, such as -Dfs.overwrite-files=true -Dtaskmanager.memory.network.min=536346624

parseDynamicProperties

public static Configuration parseDynamicProperties(org.apache.commons.cli.CommandLine cmd)

Parse the dynamic properties (passed on the command line).

getTaskManagerShellCommand

public static String getTaskManagerShellCommand(Configuration flinkConfig,
                                                ContaineredTaskManagerParameters tmParams,
                                                String configDirectory,
                                                String logDirectory,
                                                boolean hasLogback,
                                                boolean hasLog4j,
                                                boolean hasKrb5,
                                                Class<?> mainClass,
                                                String mainArgs)

Generates the shell command to start a task manager.

Parameters:

flinkConfig - The Flink configuration.

tmParams - Parameters for the task manager.

configDirectory - The configuration directory for the flink-conf.yaml

logDirectory - The log directory.

hasLogback - Uses logback?

hasLog4j - Uses log4j?

mainClass - The main class to start with.

Returns:

A String containing the task manager startup command.

getStartCommand

public static String getStartCommand(String template,
                                     Map<String,String> startCommandValues)

Replaces placeholders in the template start command with values from startCommandValues.

If the default template ConfigConstants.DEFAULT_YARN_CONTAINER_START_COMMAND_TEMPLATE is used, the following keys must be present in the map or the resulting command will still contain placeholders:

• java = path to the Java executable
• jvmmem = JVM memory limits and tweaks
• jvmopts = misc options for the Java VM
• logging = logging-related configuration settings
• class = main class to execute
• args = arguments for the main class
• redirects = output redirects

Parameters:

template - a template start command with placeholders

startCommandValues - a replacement map placeholder -> value

Returns:

the start command with placeholders filled in

updateTmpDirectoriesInConfiguration

public static void updateTmpDirectoriesInConfiguration(Configuration configuration,
                                                       @Nullable
                                                       String defaultDirs)

Set temporary configuration directories if necessary.

Parameters:

configuration - flink config to patch

defaultDirs - in case no tmp directories is set, next directories will be applied

cloneConfiguration

public static Configuration cloneConfiguration(Configuration configuration)

Clones the given configuration and resets instance specific config options.

Parameters:

configuration - to clone

Returns:

Cloned configuration with reset instance specific config options

getDynamicPropertiesAsString

public static String getDynamicPropertiesAsString(Configuration baseConfig,
                                                  Configuration targetConfig)

Get dynamic properties based on two Flink configurations. If base config does not contain and target config contains the key or the value is different, it should be added to results. Otherwise, if the base config contains and target config does not contain the key, it will be ignored.

Parameters:

baseConfig - The base configuration.

targetConfig - The target configuration.

Returns:

Dynamic properties as string, separated by whitespace.

escapeForDifferentOS

public static String escapeForDifferentOS(String value)

Escape all the dynamic property values. For Unix-like OS(Linux, MacOS, FREE_BSD, etc.), each value will be surrounded with single quotes. This works for all chars except single quote itself. To escape the single quote, close the quoting before it, insert the escaped single quote, and then re-open the quoting. For example, the value is foo'bar and the escaped value is 'foo'\''bar'. See https://stackoverflow.com/questions/15783701/which-characters-need-to-be-escaped-when-using-bash for more information about Unix escaping.

For Windows OS, each value will be surrounded with double quotes. The double quote itself needs to be escaped with back slash. Also the caret symbol need to be escaped with double carets since Windows uses it to escape characters. See https://en.wikibooks.org/wiki/Windows_Batch_Scripting for more information about Windows escaping.

Parameters:

value - value to be escaped

Returns:

escaped value

escapeWithSingleQuote

public static String escapeWithSingleQuote(String value)

escapeWithDoubleQuote

public static String escapeWithDoubleQuote(String value)

calculateHeapSize

public static int calculateHeapSize(int memory,
                                    Configuration conf)

Calculate heap size after cut-off. The heap size after cut-off will be used to set -Xms and -Xmx for jobmanager start command.

getEnvironmentVariables

public static Map<String,String> getEnvironmentVariables(String envPrefix,
                                                         Configuration flinkConfiguration)

Method to extract environment variables from the flinkConfiguration based on the given prefix String.

Parameters:

envPrefix - Prefix for the environment variables key

flinkConfiguration - The Flink config to get the environment variable definition from