A factory for components of general purpose test jobs for Flink's DataStream API operators and primitives.
The components can be configured for different state backends, including memory, file, and RocksDB
state backends. It also allows specifying the processing guarantee semantics, which will also be verified
by the job itself according to the specified semantic.
Program parameters:
- test.semantics (String, default - 'exactly-once'): This configures the semantics to test. Can be 'exactly-once' or 'at-least-once'.
- test.simulate_failure (boolean, default - false): This configures whether or not to simulate failures by throwing exceptions within the job.
- test.simulate_failure.num_records (long, default - 100L): The number of records to process before throwing an exception, per job execution attempt.
Only relevant if configured to simulate failures.
- test.simulate_failure.num_checkpoints (long, default - 1L): The number of complete checkpoints before throwing an exception, per job execution attempt.
Only relevant if configured to simulate failures.
- test.simulate_failure.max_failures (int, default - 1): The maximum number of times to fail the job. This also takes into account failures that
were not triggered by the job's own failure simulation, e.g. TaskManager or JobManager failures. Only relevant if configured to simulate failures.
- environment.checkpoint_interval (long, default - 1000): the checkpoint interval.
- environment.externalize_checkpoint (boolean, default - false): whether or not checkpoints should be externalized.
- environment.externalize_checkpoint.cleanup (String, default - 'retain'): Configures the cleanup mode for externalized checkpoints. Can be 'retain' or 'delete'.
- environment.parallelism (int, default - 1): parallelism to use for the job.
- environment.max_parallelism (int, default - 128): max parallelism to use for the job
- environment.restart_strategy (String, default - 'fixed_delay'): The failure restart strategy to use. Can be 'fixed_delay' or 'no_restart'.
- environment.restart_strategy.fixed_delay.attempts (Integer, default - Integer.MAX_VALUE): The number of allowed attempts to restart the job, when using 'fixed_delay' restart.
- environment.restart_strategy.fixed_delay.delay (long, default - 0): delay between restart attempts, in milliseconds, when using 'fixed_delay' restart.
- state_backend (String, default - 'file'): Supported values are 'file' for FsStateBackend and 'rocks' for RocksDBStateBackend.
- state_backend.checkpoint_directory (String): The checkpoint directory.
- state_backend.rocks.incremental (boolean, default - false): Activate or deactivate incremental snapshots if RocksDBStateBackend is selected.
- state_backend.file.async (boolean, default - true): Activate or deactivate asynchronous snapshots if FileStateBackend is selected.
- sequence_generator_source.keyspace (int, default - 1000): Number of different keys for events emitted by the sequence generator.
- sequence_generator_source.payload_size (int, default - 20): Length of message payloads emitted by the sequence generator.
- sequence_generator_source.sleep_time (long, default - 0): Milliseconds to sleep after emitting events in the sequence generator. Set to 0 to disable sleeping.
- sequence_generator_source.sleep_after_elements (long, default - 0): Number of elements to emit before sleeping in the sequence generator. Set to 0 to disable sleeping.
- sequence_generator_source.event_time.max_out_of_order (long, default - 500): Max event time out-of-orderness for events emitted by the sequence generator.
- sequence_generator_source.event_time.clock_progress (long, default - 100): The amount of event time to progress per event generated by the sequence generator.
- tumbling_window_operator.num_events (long, default - 20L): The duration of the window, indirectly determined by the target number of events in each window.
Total duration is (sliding_window_operator.num_events) * (sequence_generator_source.event_time.clock_progress).
- test_slide_factor (int, default - 3): test_slide_factor (int, default - 3): how many slides there are in a
single window (in other words at most how many windows may be opened at time for a given key) The length of
a window will be calculated as (test_slide_size) * (test_slide_factor)
- test_slide_size (long, default - 250): length of a slide of sliding window in milliseconds. The length of a window will be calculated as (test_slide_size) * (test_slide_factor)