Quick Start #

This document provides a quick introduction to using the Flink Kubernetes Operator. Readers of this document will be able to deploy the Flink operator itself and an example Flink job to a local Kubernetes installation.

Prerequisites #

We assume that you have a local installations of the following:

So that the kubectl and helm commands are available on your local system.

For docker we recommend that you have Docker Desktop installed and configured with at least 8GB of RAM. For kubernetes minikube is our choice, at the time of writing this we are using version v1.25.3 (end-to-end tests are using the same version). You can start a cluster with the following command:

minikube start --kubernetes-version=v1.25.3
😄  minikube v1.28.0 on Darwin 13.0.1
✨  Automatically selected the docker driver. Other choices: hyperkit, ssh
📌  Using Docker Desktop driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=4000MB) ...
🐳  Preparing Kubernetes v1.25.3 on Docker 20.10.20 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

We also recommend k9s as GUI for kubernetes, but it is optional for this quickstart guide.

Deploying the operator #

Install the certificate manager on your Kubernetes cluster to enable adding the webhook component (only needed once per Kubernetes cluster):

kubectl create -f https://github.com/jetstack/cert-manager/releases/download/v1.8.2/cert-manager.yaml

In case the cert manager installation failed for any reason you can disable the webhook by passing --set webhook.create=false to the helm install command for the operator.

Now you can deploy the selected stable Flink Kubernetes Operator version using the included Helm chart:

helm repo add flink-operator-repo https://downloads.apache.org/flink/flink-kubernetes-operator-1.4.0/
helm install flink-kubernetes-operator flink-operator-repo/flink-kubernetes-operator

To find the list of stable versions please visit https://flink.apache.org/downloads.html

The Helm chart by default points to the ghcr.io/apache/flink-kubernetes-operator image repository. If you have connectivity issues or prefer to use Dockerhub instead you can use --set image.repository=apache/flink-kubernetes-operator during installation.

You may verify your installation via kubectl and helm:

kubectl get pods
NAME READY STATUS RESTARTS AGE
flink-kubernetes-operator-fb5d46f94-ghd8b 2/2 Running 0 4m21s

helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
flink-kubernetes-operator default 1 2022-03-09 17 (tel:12022030917):39:55.461359 +0100 CET deployed flink-kubernetes-operator-1.4.0 1.4.0

Submitting a Flink job #

Once the operator is running as seen in the previous step you are ready to submit a Flink job:

kubectl create -f https://raw.githubusercontent.com/apache/flink-kubernetes-operator/release-1.4/examples/basic.yaml

You may follow the logs of your job, after a successful startup (which can take on the order of a minute in a fresh environment, seconds afterwards) you can:

kubectl logs -f deploy/basic-example

2022-03-11 21:46:04,458 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 206 (type=CHECKPOINT) @ 1647035164458 for job a12c04ac7f5d8418d8ab27931bf517b7.
2022-03-11 21:46:04,465 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 206 for job a12c04ac7f5d8418d8ab27931bf517b7 (28509 bytes, checkpointDuration=7 ms, finalizationTime=0 ms).
2022-03-11 21:46:06,458 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 207 (type=CHECKPOINT) @ 1647035166458 for job a12c04ac7f5d8418d8ab27931bf517b7.
2022-03-11 21:46:06,483 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 207 for job a12c04ac7f5d8418d8ab27931bf517b7 (28725 bytes, checkpointDuration=25 ms, finalizationTime=0 ms).

To expose the Flink Dashboard you may add a port-forward rule or look the ingress configuration options:

kubectl port-forward svc/basic-example-rest 8081

Now the Flink Dashboard is accessible at localhost:8081.

In order to stop your job and delete your FlinkDeployment you can:

kubectl delete flinkdeployment/basic-example