public class ReadWriteUtils extends Object
Modifier and Type | Field and Description |
---|---|
static org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper |
OBJECT_MAPPER |
Constructor and Description |
---|
ReadWriteUtils() |
Modifier and Type | Method and Description |
---|---|
static Stage<?> |
loadGraph(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment env,
String path,
String expectedClassName)
Loads a Graph or GraphModel from the given path.
|
static Map<String,?> |
loadMetadata(String path,
String expectedClassName)
Loads the metadata from the metadata file under the given path.
|
static <T> org.apache.flink.streaming.api.datastream.DataStream<T> |
loadModelData(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment env,
String path,
org.apache.flink.connector.file.src.reader.SimpleStreamFormat<T> modelDecoder)
Loads the model data from the given path using the model decoder.
|
static List<Stage<?>> |
loadPipeline(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment env,
String path,
String expectedClassName)
Loads the stages of a Pipeline or PipelineModel from the given path.
|
static Stage<?> |
loadStage(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment env,
String path)
Loads the stage from the given path by invoking the static load() method of the stage.
|
static <T extends Stage<T>> |
loadStageParam(String path)
Loads the stage with the saved parameters from the given path.
|
static void |
saveGraph(Stage<?> graph,
GraphData graphData,
String path)
Saves a Graph or GraphModel with the given GraphData to the given path.
|
static void |
saveMetadata(Stage<?> stage,
String path)
Saves the metadata of the given stage to a file named `metadata` under the given path.
|
static void |
saveMetadata(Stage<?> stage,
String path,
Map<String,?> extraMetadata)
Saves the metadata of the given stage and the extra metadata to a file named `metadata` under
the given path.
|
static <T> void |
saveModelData(org.apache.flink.streaming.api.datastream.DataStream<T> model,
String path,
org.apache.flink.api.common.serialization.Encoder<T> modelEncoder)
Saves the model data stream to the given path using the model encoder.
|
static void |
savePipeline(Stage<?> pipeline,
List<Stage<?>> stages,
String path)
Saves a Pipeline or PipelineModel with the given list of stages to the given path.
|
static <T> void |
setParam(Stage<?> stage,
Param<T> param,
Object value) |
static void |
updateExistingParams(Stage<?> stage,
Map<Param<?>,Object> paramOverrides) |
public static final org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper OBJECT_MAPPER
public static void saveMetadata(Stage<?> stage, String path, Map<String,?> extraMetadata) throws IOException
Required: the metadata file under the given path should not exist.
stage
- The stage instance.path
- The parent directory to save the stage metadata.extraMetadata
- The extra metadata to be saved.IOException
public static void saveMetadata(Stage<?> stage, String path) throws IOException
Required: the metadata file under the given path should not exist.
stage
- The stage instance.path
- The parent directory to save the stage metadata.IOException
public static Map<String,?> loadMetadata(String path, String expectedClassName) throws IOException
The method throws RuntimeException if the expectedClassName is not empty AND it does not match the className of the previously saved stage.
path
- The parent directory of the metadata file to read from.expectedClassName
- The expected class name of the stage.IOException
public static void savePipeline(Stage<?> pipeline, List<Stage<?>> stages, String path) throws IOException
pipeline
- A Pipeline or PipelineModel instance.stages
- A list of stages of the given pipeline.path
- The parent directory to save the pipeline metadata and its stages.IOException
public static List<Stage<?>> loadPipeline(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment env, String path, String expectedClassName) throws IOException
The method throws RuntimeException if the expectedClassName is not empty AND it does not match the className of the previously saved Pipeline or PipelineModel.
env
- A StreamExecutionEnvironment instance.path
- The parent directory to load the pipeline metadata and its stages.expectedClassName
- The expected class name of the pipeline.IOException
public static void saveGraph(Stage<?> graph, GraphData graphData, String path) throws IOException
graph
- A Graph or GraphModel instance.graphData
- A GraphData instance.path
- The parent directory to save the graph metadata and its stages.IOException
public static Stage<?> loadGraph(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment env, String path, String expectedClassName) throws IOException
The method throws RuntimeException if the expectedClassName is not empty AND it does not match the className of the previously saved Pipeline or PipelineModel.
env
- A StreamExecutionEnvironment instance.path
- The parent directory to load the pipeline metadata and its stages.expectedClassName
- The expected class name of the pipeline.IOException
public static void updateExistingParams(Stage<?> stage, Map<Param<?>,Object> paramOverrides)
public static <T extends Stage<T>> T loadStageParam(String path) throws IOException
Note: This method does not attempt to read model data from the given path. Caller needs to read model data from the given path if the stage has model data.
Required: the class with type T must have a no-argument constructor.
T
- The class type of the Stage subclass.path
- The parent directory of the stage metadata file.IOException
public static Stage<?> loadStage(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment env, String path) throws IOException
Required: the stage class must have a static load() method.
env
- A StreamExecutionEnvironment instance.path
- The parent directory of the stage metadata file.IOException
public static <T> void saveModelData(org.apache.flink.streaming.api.datastream.DataStream<T> model, String path, org.apache.flink.api.common.serialization.Encoder<T> modelEncoder)
T
- The class type of the model data.model
- The model data stream.path
- The parent directory of the model data file.modelEncoder
- The encoder to encode the model data.public static <T> org.apache.flink.streaming.api.datastream.DataStream<T> loadModelData(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment env, String path, org.apache.flink.connector.file.src.reader.SimpleStreamFormat<T> modelDecoder)
T
- The class type of the model data.env
- A StreamExecutionEnvironment instance.path
- The parent directory of the model data file.modelDecoder
- The decoder used to decode the model data.Copyright © 2019–2023 The Apache Software Foundation. All rights reserved.