This documentation is for an unreleased version of the Apache Flink Kubernetes Operator. We recommend you use the latest stable version.
Pod template #
The operator CRD is designed to have a minimal set of direct, short-hand CRD settings to express the most
basic attributes of a deployment. For all other settings the CRD provides the flinkConfiguration
and
podTemplate
fields.
Pod templates permit customization of the Flink job and task manager pods, for example to specify volume mounts, ephemeral storage, sidecar containers etc.
Pod templates can be layered, as shown in the example below.
A common pod template may hold the settings that apply to both job and task manager,
like volumeMounts
. Another template under job or task manager can define additional settings that supplement or override those
in the common template, such as a task manager sidecar.
The operator is going to merge the common and specific templates for job and task manager respectively.
Here the full example:
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
namespace: default
name: pod-template-example
spec:
image: flink:1.17
flinkVersion: v1_17
flinkConfiguration:
taskmanager.numberOfTaskSlots: "2"
serviceAccount: flink
podTemplate:
spec:
containers:
# Do not change the main container name
- name: flink-main-container
volumeMounts:
- mountPath: /opt/flink/log
name: flink-logs
# Sample sidecar container
- name: fluentbit
image: fluent/fluent-bit:1.8.12-debug
command: [ 'sh','-c','/fluent-bit/bin/fluent-bit -i tail -p path=/flink-logs/*.log -p multiline.parser=java -o stdout' ]
volumeMounts:
- mountPath: /flink-logs
name: flink-logs
volumes:
- name: flink-logs
emptyDir: { }
jobManager:
resource:
memory: "2048m"
cpu: 1
taskManager:
resource:
memory: "2048m"
cpu: 1
podTemplate:
spec:
initContainers:
# Sample sidecar container
- name: busybox
image: busybox:1.35.0
command: [ 'sh','-c','echo hello from task manager' ]
job:
jarURI: local:///opt/flink/examples/streaming/StateMachineExample.jar
parallelism: 2
When using the operator with Flink native Kubernetes integration, please refer to pod template field precedence.
Array Merging Behaviour #
When layering pod templates (defining both a top level and jobmanager specific podtemplate for example) the corresponding yamls are merged together.
The default behaviour of the pod template mechanism is to merge array arrays by merging the objects in the respective array positions. This requires that containers in the podTemplates are defined in the same order otherwise the results may be undefined.
Default behaviour (merge by position):
arr1: [{name: a, p1: v1}, {name: b, p1: v1}]
arr1: [{name: a, p2: v2}, {name: c, p2: v2}]
merged: [{name: a, p1: v1, p2: v2}, {name: c, p1: v1, p2: v2}]
The operator supports an alternative array merging mechanism that can be enabled by the kubernetes.operator.pod-template.merge-arrays-by-name
flag.
When true, instead of the default positional merging, object array elements that have a name
property defined will be merged by their name and the resulting array will be a union of the two input arrays.
Merge by name:
arr1: [{name: a, p1: v1}, {name: b, p1: v1}]
arr1: [{name: a, p2: v2}, {name: c, p2: v2}]
merged: [{name: a, p1: v1, p2: v2}, {name: b, p1: v1}, {name: c, p2: v2}]
Merging by name can be very convenient when merging container specs or when the base and override templates are not defined together.