org.apache.flink.ml.common

Class FlinkMLTools$

java.lang.Object

org.apache.flink.ml.common.FlinkMLTools$

public class FlinkMLTools$
extends Object

FlinkTools contains a set of convenience functions for Flink's machine learning library:

- persist: Takes up to 5 DataSets and file paths. Each DataSet is written to the specified path and subsequently re-read from disk. This method can be used to effectively split the execution graph at the given DataSet. Writing it to disk triggers its materialization and specifying it as a source will prevent the re-execution of it.

- block: Takes a DataSet of elements T and groups them in n blocks.

Field Summary

Fields

Modifier and Type	Field and Description
static FlinkMLTools$	MODULE$ Static reference to the singleton instance of this Scala object.

Constructor Summary

Constructors

Constructor and Description
FlinkMLTools$()

Method Summary

Modifier and Type	Method and Description
<T> DataSet<Block<T>>	block(DataSet<T> input, int numBlocks, scala.Option<Partitioner<Object>> partitionerOption, TypeInformation<T> evidence$31, scala.reflect.ClassTag<T> evidence$32) Groups the DataSet input into numBlocks blocks.
<A,B,C,D,E> scala.Tuple5<DataSet<A>,DataSet<B>,DataSet<C>,DataSet<D>,DataSet<E>>	persist(DataSet<A> ds1, DataSet<B> ds2, DataSet<C> ds3, DataSet<D> ds4, DataSet<E> ds5, String path1, String path2, String path3, String path4, String path5, scala.reflect.ClassTag<A> evidence$21, TypeInformation<A> evidence$22, scala.reflect.ClassTag<B> evidence$23, TypeInformation<B> evidence$24, scala.reflect.ClassTag<C> evidence$25, TypeInformation<C> evidence$26, scala.reflect.ClassTag<D> evidence$27, TypeInformation<D> evidence$28, scala.reflect.ClassTag<E> evidence$29, TypeInformation<E> evidence$30) Writes multiple DataSets to the specified paths and returns them as DataSources for subsequent operations.
<A,B,C,D> scala.Tuple4<DataSet<A>,DataSet<B>,DataSet<C>,DataSet<D>>	persist(DataSet<A> ds1, DataSet<B> ds2, DataSet<C> ds3, DataSet<D> ds4, String path1, String path2, String path3, String path4, scala.reflect.ClassTag<A> evidence$13, TypeInformation<A> evidence$14, scala.reflect.ClassTag<B> evidence$15, TypeInformation<B> evidence$16, scala.reflect.ClassTag<C> evidence$17, TypeInformation<C> evidence$18, scala.reflect.ClassTag<D> evidence$19, TypeInformation<D> evidence$20) Writes multiple DataSets to the specified paths and returns them as DataSources for subsequent operations.
<A,B,C> scala.Tuple3<DataSet<A>,DataSet<B>,DataSet<C>>	persist(DataSet<A> ds1, DataSet<B> ds2, DataSet<C> ds3, String path1, String path2, String path3, scala.reflect.ClassTag<A> evidence$7, TypeInformation<A> evidence$8, scala.reflect.ClassTag<B> evidence$9, TypeInformation<B> evidence$10, scala.reflect.ClassTag<C> evidence$11, TypeInformation<C> evidence$12) Writes multiple DataSets to the specified paths and returns them as DataSources for subsequent operations.
<A,B> scala.Tuple2<DataSet<A>,DataSet<B>>	persist(DataSet<A> ds1, DataSet<B> ds2, String path1, String path2, scala.reflect.ClassTag<A> evidence$3, TypeInformation<A> evidence$4, scala.reflect.ClassTag<B> evidence$5, TypeInformation<B> evidence$6) Writes multiple DataSets to the specified paths and returns them as DataSources for subsequent operations.
<T> DataSet<T>	persist(DataSet<T> dataset, String path, scala.reflect.ClassTag<T> evidence$1, TypeInformation<T> evidence$2) Writes a DataSet to the specified path and returns it as a DataSource for subsequent operations.
void	registerFlinkMLTypes(ExecutionEnvironment env) Registers the different FlinkML related types for Kryo serialization

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MODULE$

public static final FlinkMLTools$ MODULE$

Static reference to the singleton instance of this Scala object.

Constructor Detail

FlinkMLTools$

public FlinkMLTools$()

Method Detail

registerFlinkMLTypes

public void registerFlinkMLTypes(ExecutionEnvironment env)

Registers the different FlinkML related types for Kryo serialization

Parameters:

env -

persist

public <T> DataSet<T> persist(DataSet<T> dataset,
                              String path,
                              scala.reflect.ClassTag<T> evidence$1,
                              TypeInformation<T> evidence$2)

Writes a DataSet to the specified path and returns it as a DataSource for subsequent operations.

Parameters:

dataset - DataSet to write to disk

path - File path to write dataset to

Returns:

DataSet reading the just written file

persist

public <A,B> scala.Tuple2<DataSet<A>,DataSet<B>> persist(DataSet<A> ds1,
                                                         DataSet<B> ds2,
                                                         String path1,
                                                         String path2,
                                                         scala.reflect.ClassTag<A> evidence$3,
                                                         TypeInformation<A> evidence$4,
                                                         scala.reflect.ClassTag<B> evidence$5,
                                                         TypeInformation<B> evidence$6)

Writes multiple DataSets to the specified paths and returns them as DataSources for subsequent operations.

Parameters:

ds1 - First DataSet to write to disk

ds2 - Second DataSet to write to disk

path1 - Path for ds1

path2 - Path for ds2

Returns:

Tuple of DataSets reading the just written files

persist

public <A,B,C> scala.Tuple3<DataSet<A>,DataSet<B>,DataSet<C>> persist(DataSet<A> ds1,
                                                                      DataSet<B> ds2,
                                                                      DataSet<C> ds3,
                                                                      String path1,
                                                                      String path2,
                                                                      String path3,
                                                                      scala.reflect.ClassTag<A> evidence$7,
                                                                      TypeInformation<A> evidence$8,
                                                                      scala.reflect.ClassTag<B> evidence$9,
                                                                      TypeInformation<B> evidence$10,
                                                                      scala.reflect.ClassTag<C> evidence$11,
                                                                      TypeInformation<C> evidence$12)

Writes multiple DataSets to the specified paths and returns them as DataSources for subsequent operations.

Parameters:

ds1 - First DataSet to write to disk

ds2 - Second DataSet to write to disk

ds3 - Third DataSet to write to disk

path1 - Path for ds1

path2 - Path for ds2

path3 - Path for ds3

Returns:

Tuple of DataSets reading the just written files

persist

public <A,B,C,D> scala.Tuple4<DataSet<A>,DataSet<B>,DataSet<C>,DataSet<D>> persist(DataSet<A> ds1,
                                                                                   DataSet<B> ds2,
                                                                                   DataSet<C> ds3,
                                                                                   DataSet<D> ds4,
                                                                                   String path1,
                                                                                   String path2,
                                                                                   String path3,
                                                                                   String path4,
                                                                                   scala.reflect.ClassTag<A> evidence$13,
                                                                                   TypeInformation<A> evidence$14,
                                                                                   scala.reflect.ClassTag<B> evidence$15,
                                                                                   TypeInformation<B> evidence$16,
                                                                                   scala.reflect.ClassTag<C> evidence$17,
                                                                                   TypeInformation<C> evidence$18,
                                                                                   scala.reflect.ClassTag<D> evidence$19,
                                                                                   TypeInformation<D> evidence$20)

Writes multiple DataSets to the specified paths and returns them as DataSources for subsequent operations.

Parameters:

ds1 - First DataSet to write to disk

ds2 - Second DataSet to write to disk

ds3 - Third DataSet to write to disk

ds4 - Fourth DataSet to write to disk

path1 - Path for ds1

path2 - Path for ds2

path3 - Path for ds3

path4 - Path for ds4

Returns:

Tuple of DataSets reading the just written files

persist

public <A,B,C,D,E> scala.Tuple5<DataSet<A>,DataSet<B>,DataSet<C>,DataSet<D>,DataSet<E>> persist(DataSet<A> ds1,
                                                                                                DataSet<B> ds2,
                                                                                                DataSet<C> ds3,
                                                                                                DataSet<D> ds4,
                                                                                                DataSet<E> ds5,
                                                                                                String path1,
                                                                                                String path2,
                                                                                                String path3,
                                                                                                String path4,
                                                                                                String path5,
                                                                                                scala.reflect.ClassTag<A> evidence$21,
                                                                                                TypeInformation<A> evidence$22,
                                                                                                scala.reflect.ClassTag<B> evidence$23,
                                                                                                TypeInformation<B> evidence$24,
                                                                                                scala.reflect.ClassTag<C> evidence$25,
                                                                                                TypeInformation<C> evidence$26,
                                                                                                scala.reflect.ClassTag<D> evidence$27,
                                                                                                TypeInformation<D> evidence$28,
                                                                                                scala.reflect.ClassTag<E> evidence$29,
                                                                                                TypeInformation<E> evidence$30)

Writes multiple DataSets to the specified paths and returns them as DataSources for subsequent operations.

Parameters:

ds1 - First DataSet to write to disk

ds2 - Second DataSet to write to disk

ds3 - Third DataSet to write to disk

ds4 - Fourth DataSet to write to disk

ds5 - Fifth DataSet to write to disk

path1 - Path for ds1

path2 - Path for ds2

path3 - Path for ds3

path4 - Path for ds4

path5 - Path for ds5

Returns:

Tuple of DataSets reading the just written files

block

public <T> DataSet<Block<T>> block(DataSet<T> input,
                                   int numBlocks,
                                   scala.Option<Partitioner<Object>> partitionerOption,
                                   TypeInformation<T> evidence$31,
                                   scala.reflect.ClassTag<T> evidence$32)

Groups the DataSet input into numBlocks blocks.

Parameters:

input -

numBlocks - Number of Blocks

partitionerOption - Optional partitioner to control the partitioning

Returns: