org.apache.flink.ml.common.datastream

Class DataStreamUtils

java.lang.Object

org.apache.flink.ml.common.datastream.DataStreamUtils

public class DataStreamUtils
extends Object

Provides utility functions for DataStream.

Constructor Summary

Constructors

Constructor and Description
DataStreamUtils()

Method Summary

Modifier and Type	Method and Description
static org.apache.flink.streaming.api.datastream.DataStream<double[]>	allReduceSum(org.apache.flink.streaming.api.datastream.DataStream<double[]> input) Applies allReduceSum on the input data stream.
static <IN,OUT> org.apache.flink.streaming.api.datastream.DataStream<OUT>	mapPartition(org.apache.flink.streaming.api.datastream.DataStream<IN> input, org.apache.flink.api.common.functions.MapPartitionFunction<IN,OUT> func) Applies a MapPartitionFunction on a bounded data stream.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DataStreamUtils

public DataStreamUtils()

Method Detail

allReduceSum

public static org.apache.flink.streaming.api.datastream.DataStream<double[]> allReduceSum(org.apache.flink.streaming.api.datastream.DataStream<double[]> input)

Applies allReduceSum on the input data stream. The input data stream is supposed to contain one double array in each partition. The result data stream has the same parallelism as the input, where each partition contains one double array that sums all of the double arrays in the input data stream.

Note that we throw exception when one of the following two cases happen:

There exists one partition that contains more than one double array. The length of the double array is not consistent among all partitions.

Parameters:

input - The input data stream.

Returns:

The result data stream.

mapPartition

public static <IN,OUT> org.apache.flink.streaming.api.datastream.DataStream<OUT> mapPartition(org.apache.flink.streaming.api.datastream.DataStream<IN> input,
                                                                                              org.apache.flink.api.common.functions.MapPartitionFunction<IN,OUT> func)

Applies a MapPartitionFunction on a bounded data stream.

Type Parameters:

IN - The class type of the input element.

OUT - The class type of output element.

Parameters:

input - The input data stream.

func - The user defined mapPartition function.

Returns:

The result data stream.