History Server #
Flink has a history server that can be used to query the statistics of completed jobs after the corresponding Flink cluster has been shut down.
Furthermore, it exposes a REST API that accepts HTTP requests and responds with JSON data.
The HistoryServer allows you to query the status and statistics of completed jobs that have been archived by a JobManager.
After you have configured the HistoryServer and JobManager, you start and stop the HistoryServer via its corresponding startup script:
# Start or stop the HistoryServer bin/historyserver.sh (start|start-foreground|stop)
By default, this server binds to
localhost and listens at port
Currently, you can only run it as a standalone process.
The configuration keys
historyserver.archive.fs.refresh-interval need to be adjusted for archiving and displaying archived jobs.
The archiving of completed jobs happens on the JobManager, which uploads the archived job information to a file system directory. You can configure the directory to archive completed jobs in
flink-conf.yaml by setting a directory via
# Directory to upload completed job information jobmanager.archive.fs.dir: hdfs:///completed-jobs
The HistoryServer can be configured to monitor a comma-separated list of directories in via
historyserver.archive.fs.dir. The configured directories are regularly polled for new archives; the polling interval can be configured via
# Monitor the following directories for completed jobs historyserver.archive.fs.dir: hdfs:///completed-jobs # Refresh every 10 seconds historyserver.archive.fs.refresh-interval: 10000
The contained archives are downloaded and cached in the local filesystem. The local directory for this is configured via
Check out the configuration page for a complete list of configuration options.
Log Integration #
Flink does not provide built-in methods for archiving logs of completed jobs.
However, if you already have log archiving and browsing services, you can configure HistoryServer to integrate them
In this way, you can directly link from HistoryServer WebUI to logs of the relevant JobManager / TaskManagers.
# HistoryServer will replace <jobid> with the relevant job id historyserver.log.jobmanager.url-pattern: http://my.log-browsing.url/<jobid> # HistoryServer will replace <jobid> and <tmid> with the relevant job id and taskmanager id historyserver.log.taskmanager.url-pattern: http://my.log-browsing.url/<jobid>/<tmid>
Available Requests #
Below is a list of available requests, with a sample JSON response. All requests are of the sample form
http://hostname:8082/jobs, below we list only the path part of the URLs.
Values in angle brackets are variables, for example
http://hostname:port/jobs/<jobid>/exceptions will have to requested for example as