org.apache.flink.streaming.python.api.environment

Class PythonEnvironmentFactory

java.lang.Object

org.apache.flink.streaming.python.api.environment.PythonEnvironmentFactory

public class PythonEnvironmentFactory
extends Object

A factory for PythonStreamExecutionEnvironments.

This class is a replacement for static factory methods defined in StreamExecutionEnvironment and allows us to pass state from the PythonStreamBinder instance to the created execution environment without having to rely on static fields.

Constructor Summary

Constructors

Constructor and Description
PythonEnvironmentFactory(String localTmpPath, Path tmpDistributedDir, String scriptName)

Method Summary

Modifier and Type	Method and Description
PythonStreamExecutionEnvironment	create_local_execution_environment(Configuration config) Creates a LocalStreamEnvironment.
PythonStreamExecutionEnvironment	create_local_execution_environment(int parallelism, Configuration config) A thin wrapper layer over StreamExecutionEnvironment.createLocalEnvironment(int, Configuration).
PythonStreamExecutionEnvironment	create_remote_execution_environment(String host, int port, Configuration config, String... jar_files) A thin wrapper layer over StreamExecutionEnvironment.createRemoteEnvironment( String, int, Configuration, String...).
PythonStreamExecutionEnvironment	create_remote_execution_environment(String host, int port, int parallelism, String... jar_files) A thin wrapper layer over StreamExecutionEnvironment.createRemoteEnvironment( String, int, int, String...).
PythonStreamExecutionEnvironment	create_remote_execution_environment(String host, int port, String... jar_files) A thin wrapper layer over StreamExecutionEnvironment.createRemoteEnvironment(String, int, String...).
PythonStreamExecutionEnvironment	get_execution_environment() A thin wrapper layer over StreamExecutionEnvironment.getExecutionEnvironment().

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PythonEnvironmentFactory

public PythonEnvironmentFactory(String localTmpPath,
                                Path tmpDistributedDir,
                                String scriptName)

Method Detail

get_execution_environment

public PythonStreamExecutionEnvironment get_execution_environment()

A thin wrapper layer over StreamExecutionEnvironment.getExecutionEnvironment(). In addition it takes care for required Jython serializers registration.

Returns:

The python execution environment of the context in which the program is executed.

create_local_execution_environment

public PythonStreamExecutionEnvironment create_local_execution_environment(Configuration config)

Creates a LocalStreamEnvironment. The local execution environment will run the program in a multi-threaded fashion in the same JVM as the environment was created in. The default parallelism of the local environment is the number of hardware contexts (CPU cores / threads), unless it was specified differently by PythonStreamExecutionEnvironment.set_parallelism(int).

Parameters:

config - Pass a custom configuration into the cluster

Returns:

A local execution environment with the specified parallelism.

create_local_execution_environment

public PythonStreamExecutionEnvironment create_local_execution_environment(int parallelism,
                                                                           Configuration config)

A thin wrapper layer over StreamExecutionEnvironment.createLocalEnvironment(int, Configuration).

Parameters:

parallelism - The parallelism for the local environment.

config - Pass a custom configuration into the cluster

Returns:

A local python execution environment with the specified parallelism.

create_remote_execution_environment

public PythonStreamExecutionEnvironment create_remote_execution_environment(String host,
                                                                            int port,
                                                                            String... jar_files)

A thin wrapper layer over StreamExecutionEnvironment.createRemoteEnvironment(String, int, String...).

Parameters:

host - The host name or address of the master (JobManager), where the program should be executed.

port - The port of the master (JobManager), where the program should be executed.

jar_files - The JAR files with code that needs to be shipped to the cluster. If the program uses user-defined functions, user-defined input formats, or any libraries, those must be provided in the JAR files.

Returns:

A remote environment that executes the program on a cluster.

create_remote_execution_environment

public PythonStreamExecutionEnvironment create_remote_execution_environment(String host,
                                                                            int port,
                                                                            Configuration config,
                                                                            String... jar_files)

A thin wrapper layer over StreamExecutionEnvironment.createRemoteEnvironment( String, int, Configuration, String...).

Parameters:

host - The host name or address of the master (JobManager), where the program should be executed.

port - The port of the master (JobManager), where the program should be executed.

config - The configuration used by the client that connects to the remote cluster.

jar_files - The JAR files with code that needs to be shipped to the cluster. If the program uses user-defined functions, user-defined input formats, or any libraries, those must be provided in the JAR files.

Returns:

A remote environment that executes the program on a cluster.

create_remote_execution_environment

public PythonStreamExecutionEnvironment create_remote_execution_environment(String host,
                                                                            int port,
                                                                            int parallelism,
                                                                            String... jar_files)

A thin wrapper layer over StreamExecutionEnvironment.createRemoteEnvironment( String, int, int, String...).

Parameters:

host - The host name or address of the master (JobManager), where the program should be executed.

port - The port of the master (JobManager), where the program should be executed.

parallelism - The parallelism to use during the execution.

jar_files - The JAR files with code that needs to be shipped to the cluster. If the program uses user-defined functions, user-defined input formats, or any libraries, those must be provided in the JAR files.

Returns:

A remote environment that executes the program on a cluster.