Interface MapPartitionFunction<T,​O>

  • Type Parameters:
    T - Type of the input elements.
    O - Type of the returned elements.
    All Superinterfaces:
    Function, Serializable
    All Known Implementing Classes:
    AssignRangeIndex, RangeBoundaryBuilder, RichMapPartitionFunction
    Functional Interface:
    This is a functional interface and can therefore be used as the assignment target for a lambda expression or method reference.

    @Public
    @FunctionalInterface
    public interface MapPartitionFunction<T,​O>
    extends Function, Serializable
    Interface for "mapPartition" functions. A "mapPartition" function is called a single time per data partition receives an Iterable with data elements of that partition. It may return an arbitrary number of data elements.

    This function is intended to provide enhanced flexibility in the processing of elements in a partition. For most of the simple use cases, consider using the MapFunction or FlatMapFunction.

    The basic syntax for a MapPartitionFunction is as follows:

    
     DataSet<X> input = ...;
    
     DataSet<Y> result = input.mapPartition(new MyMapPartitionFunction());
     
    • Method Detail

      • mapPartition

        void mapPartition​(Iterable<T> values,
                          Collector<O> out)
                   throws Exception
        A user-implemented function that modifies or transforms an incoming object.
        Parameters:
        values - All records for the mapper
        out - The collector to hand results to.
        Throws:
        Exception - This method may throw exceptions. Throwing an exception will cause the operation to fail and may trigger recovery.