This Getting Started section guides you through setting up a fully functional Flink Cluster on Mesos.
Apache Mesos is another resource provider supported by
Apache Flink. Flink utilizes the worker’s provided by Mesos to run its TaskManagers.
Apache Flink provides the script bin/mesos-appmaster.sh
to initiate the Flink
on Mesos cluster.
Flink on Mesos expects a Mesos cluster to be around. It also requires the Flink binaries being deployed. Additionally, Hadoop needs to be installed on the very same machine.
Flink provides bin/mesos-appmaster.sh
to initiate a Flink on Mesos cluster. A Mesos application master
will be created (i.e. a JobManager process with Mesos support) which will utilize the Mesos workers to
run Flink’s TaskManager processes.
For bin/mesos-appmaster.sh
to work, you have to set the two variables HADOOP_CLASSPATH
and
MESOS_NATIVE_JAVA_LIBRARY
:
MESOS_NATIVE_JAVA_LIBRARY
needs to point to Mesos’ native Java library. The library name libmesos.so
used above refers to Mesos’ Linux library. Running Mesos on MacOS would require you to use
libmesos.dylib
instead.
Connect to the machine which matches all the requirements listed in the Preparation section.
Change into Flink’s home directory and call bin/mesos-appmaster.sh
:
The commands above use a few placeholders that need to be substituted by settings of the actual underlying cluster:
<mesos-master>
refers to the Mesos master’s IP address or hostname.<jobmanager-host>
refers to the host that executes bin/mesos-appmaster.sh
which is starting
Flink’s JobManager process. It’s important to not use localhost
or 127.0.0.1
as this parameter
is being shared with the Mesos cluster and the TaskManagers.<flink-user>
refers to the user that owns the Mesos master’s Flink installation directory (see Mesos’
documentation on specifying a user
for further details).The run
action requires --target
to be set to remote
. Refer to the CLI documentation
for further details on that parameter.
The Flink on Mesos cluster is now deployed in Session Mode. Note that you can run multiple Flink jobs on a Session cluster. Each job needs to be submitted to the cluster. TaskManagers are deployed on the Mesos workers as needed. Keep in mind that you can only run as many jobs as the Mesos cluster allows in terms of resources provided by the Mesos workers. Play around with Flink’s parameters to find the right resource utilization for your needs.
Check out Flink’s Mesos configuration to further influence the resources Flink on Mesos is going to allocate.
For production use, we recommend deploying Flink Applications in the Per-Job Mode, as it provides a better isolation for each job.
Flink on Mesos does not support Application Mode.
A job which is executed in Per-Job Cluster Mode spins
up a dedicated Flink cluster that is only used for that specific job. No extra job submission is
needed. bin/mesos-appmaster-job.sh
is used as the startup script. It will start a Flink cluster
for a dedicated job which is passed as a JobGraph file. This file can be created by applying the
following code to your Job source code:
Flink on Mesos Per-Job cluster can be started in the following way:
<job-graph-file>
refers to the path of the uploaded JobGraph file defining the job that shall be
executed on the Per-Job Flink cluster in the command above. The meaning of <mesos-master>
,
<jobmanager-host>
and <flink-user>
are described in the
Getting Started guide of this page.
The Getting Started guide at the top of this page describes deploying Flink in Session Mode.
User libraries can be passed to the Mesos workers by placing them in Flink’s lib/
folder. This way,
they will be picked by Mesos’ Fetcher and copied over into the worker’s sandbox folders. Alternatively,
Docker containerization can be used as described in Installing Flink on the Workers.
Flink on Mesos offers two ways to distribute the Flink and user binaries within the Mesos cluster:
mesos
. This is the default option and is used in the example commands of this page.docker
and by providing the image name through mesos.resourcemanager.tasks.container.image.name.You will need to run a service like Marathon or Apache Aurora which takes care of restarting the JobManager process in case of node or process failures. In addition, Zookeeper needs to be configured as described in the High Availability section of the Flink docs.
Marathon needs to be set up to launch the bin/mesos-appmaster.sh
script. In particular, it should
also adjust any configuration parameters for the Flink cluster.
Here is an example configuration for Marathon:
Flink is installed into /opt/flink-1.12.7
having <flink-user>
as the owner of the Flink
directory (notice that the user is used twice: once as a Marathon and another time as a Mesos
parameter) for the example configuration above to work. Additionally, we have the bundled Hadoop jar
saved in Flink’s lib/
folder for the sake of simplicity here. This way, we don’t have to set
HADOOP_CLASSPATH
as a environment variable next to MESOS_NATIVE_JAVA_LIBRARY
.
<mesos-master>
needs to be set to the hostname or IP of Mesos’ master node. $HOST
is a Marathon
environment variable referring to the hostname of the machine the script is executed on. $HOST
should
not be replaced in the config above!
The whole Flink cluster including the JobManager will be run as Mesos tasks in the Mesos cluster when deploying Flink using Marathon. Flink’s binaries have to be installed on all Mesos workers for the above Marathon config to work.
Flink on Mesos is compiled against Hadoop 2.4.1, and all Hadoop versions >= 2.4.1 are supported, including Hadoop 3.x.
For providing Flink with the required Hadoop dependencies, we recommend setting the HADOOP_CLASSPATH
environment variable already introduced in the Getting Started / Preparation section.
If that is not possible, the dependencies can also be put into the lib/
folder of Flink.
Flink also offers pre-bundled Hadoop fat jars for placing them in the lib/
folder, on the
Downloads / Additional Components section of the website.
These pre-bundled fat jars are shaded to avoid dependency conflicts with common libraries. The Flink
community is not testing the Mesos integration against these pre-bundled jars.
The Flink on Mesos implementation consists of two components: The application master and the workers. The workers are simple TaskManagers parameterized by the environment which is set up through the application master. The most sophisticated component of the Flink on Mesos implementation is the application master. The application master currently hosts the following components:
Mesos Scheduler: The Scheduler is responsible for registering a framework with Mesos, requesting resources, and launching worker nodes. The Scheduler continuously needs to report back to Mesos to ensure the framework is in a healthy state. To verify the health of the cluster, the Scheduler monitors the spawned workers, marks them as failed and restarts them if necessary.
Flink’s Mesos Scheduler itself is currently not highly available. However, it persists all necessary information about its state (e.g. configuration, list of workers) in ZooKeeper. In the presence of a failure, it relies on an external system to bring up a new Scheduler (see the Marathon subsection for further details). The Scheduler will then register with Mesos again and go through the reconciliation phase. In the reconciliation phase, the Scheduler receives a list of running workers nodes. It matches these against the recovered information from ZooKeeper and makes sure to bring back the cluster in the state before the failure.
Artifact Server: The Artifact Server is responsible for providing resources to the worker nodes. The resources can be anything from the Flink binaries to shared secrets or configuration files. For instance, in non-containerized environments, the Artifact Server will provide the Flink binaries. What files will be served depends on the configuration overlay used.
Flink’s Mesos startup scripts bin/mesos-appmaster.sh
and bin/mesos-appmaster-job.sh
provide a way
to configure and start the application master. The worker nodes inherit all further configuration.
They are deployed through bin/mesos-taskmanager.sh
. The configuration inheritance is achieved using
configuration overlays. Configuration overlays provide a way to infer a configuration from environment
variables and config files which are shipped to the worker nodes.
See Mesos Architecture for a more details on how frameworks are handled by Mesos.
The following resource files can be used to set up a local Mesos cluster running the Marathon framework and having Flink 1.11.2 installed.
The docker-compose.yml
provided below is based on the work done by
Sean Bennet.
Keep in mind that it requires the Dockerfile
of the previous section to be found in the same
directory and the file being named Dockerfile
. It might make sense to scale the worker nodes up to
have enough workers to run Flink on Mesos next
to the Marathon framework:
The following Marathon configuration can be applied through the Marathon UI: http://localhost:8080/
It will start a Flink on Mesos cluster on any of the worker machines. Flink’s default port 8081
is
forwarded to random ports due to the scaling of the worker nodes. Use docker ps
to figure out the
host system’s ports that to be able to access Flink’s web interface.