This documentation is for an out-of-date version of Apache Flink. We recommend you use the latest stable version.
  1. Deployment & Operations
  2. Clusters & Deployment
  3. Docker

Docker Setup

Docker is a popular container runtime. There are Docker images for Apache Flink available on Docker Hub which can be used to deploy a session cluster. The Flink repository also contains tooling to create container images to deploy a job cluster.

A Flink session cluster can be used to run multiple jobs. Each job needs to be submitted to the cluster after it has been deployed.

Docker images

The Flink Docker repository is hosted on Docker Hub and serves images of Flink version 1.2.1 and later.

Images for each supported combination of Hadoop and Scala are available, and tag aliases are provided for convenience.

Beginning with Flink 1.5, image tags that omit a Hadoop version (e.g. -hadoop28) correspond to Hadoop-free releases of Flink that do not include a bundled Hadoop distribution.

For example, the following aliases can be used: (1.5.y indicates the latest release of Flink 1.5)

  • flink:latest → flink:<latest-flink>-scala_<latest-scala>
  • flink:1.5 → flink:1.5.y-scala_2.11
  • flink:1.5-hadoop27 → flink:1.5.y-hadoop27-scala_2.11

Note: The Docker images are provided as a community project by individuals on a best-effort basis. They are not official releases by the Apache Flink PMC.

A Flink job cluster is a dedicated cluster which runs a single job. The job is part of the image and, thus, there is no extra job submission needed.

Docker images

The Flink job cluster image needs to contain the user code jars of the job for which the cluster is started. Therefore, one needs to build a dedicated container image for every job. The flink-container module contains a build.sh script which can be used to create such an image. Please see the instructions for more details.

Docker Compose is a convenient way to run a group of Docker containers locally.

Example config files for a session cluster and a job cluster are available on GitHub.

Usage

  • Launch a cluster in the foreground

      docker-compose up

  • Launch a cluster in the background

      docker-compose up -d

  • Scale the cluster up or down to N TaskManagers

      docker-compose scale taskmanager=<N>

  • Kill the cluster

      docker-compose kill

When the cluster is running, you can visit the web UI at http://localhost:8081. You can also use the web UI to submit a job to a session cluster.

To submit a job to a session cluster via the command line, you must copy the JAR to the JobManager container and submit the job from there.

For example:

$ JOBMANAGER_CONTAINER=$(docker ps --filter name=jobmanager --format={{.ID}})
$ docker cp path/to/jar "$JOBMANAGER_CONTAINER":/job.jar
$ docker exec -t -i "$JOBMANAGER_CONTAINER" flink run /job.jar

Back to top