This documentation is for an out-of-date version of Apache Flink. We recommend you use the latest stable version.
Execution Mode #
The Python API supports different runtime execution modes from which you can choose depending on the requirements of your use case and the characteristics of your job. The Python runtime execution mode defines how to execute your customized Python functions.
Prior to release-1.15, there is the only execution mode called PROCESS
execution mode. The PROCESS
mode means that the Python user-defined functions will be executed in separate Python processes.
In release-1.15, it has introduced a new execution mode called THREAD
execution mode. The THREAD
mode means that the Python user-defined functions will be executed in the same process as Java Operator,
It should be noted that multiple Python user-defined functions running in the same JVM are still affected by GIL.
When can/should I use THREAD execution mode? #
The purpose of the introduction of THREAD
mode is to overcome the overhead of serialization/deserialization
and network communication caused in PROCESS
mode. So if performance is not your concern, or the computing
logic of your customized Python functions is the performance bottleneck of the job, PROCESS
mode will
be the best choice as PROCESS
mode provides the best isolation compared to THREAD
mode.
Configuring Python execution mode #
The execution mode can be configured via the python.execution-mode
setting.
There are two possible values:
PROCESS
: The Python user-defined functions will be executed in separate Python process. (default)THREAD
: The Python user-defined functions will be executed in the same process as Java operator.
You could specify the Python execution mode using Python Table API as following:
# Specify `PROCESS` mode
table_env.get_config().set("python.execution-mode", "process")
# Specify `THREAD` mode
table_env.get_config().set("python.execution-mode", "thread")
Currently, it still doesn’t support to execute Python UDFs inTHREAD
execution mode in all places. It will fall back toPROCESS
execution mode in these cases. So it may happen that you configure a job to execute inTHREAD
execution mode, however, it’s actually executed inPROCESS
execution mode.
THREAD
execution mode is only supported in Python 3.7+.
Execution Behavior #
This section provides an overview of the execution behavior of THREAD
execution mode and contrasts
they with PROCESS
execution mode. For more details, please refer to the FLIP that introduced this feature:
FLIP-206.
PROCESS Execution Mode #
In PROCESS
execution mode, the Python user-defined functions will be executed in separate Python Worker process.
The Java operator process communicates with the Python worker process using various Grpc services.

THREAD Execution Mode #
In THREAD
execution mode, the Python user-defined functions will be executed in the same process
as Java operators. PyFlink takes use of third part library PEMJA
to embed Python in Java Application.
