org.apache.flink.python.api.streaming.data

Class PythonStreamer

java.lang.Object

org.apache.flink.python.api.streaming.data.PythonStreamer

All Implemented Interfaces:

Serializable

public class PythonStreamer
extends Object
implements Serializable

This streamer is used by functions to send/receive data to/from an external python process.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected AbstractRichFunction	function
protected DataInputStream	in
protected static org.slf4j.Logger	LOG
protected StringBuilder	msg
protected DataOutputStream	out
protected int	port
protected PythonReceiver	receiver
protected PythonSender	sender
protected ServerSocket	server
protected Socket	socket

Constructor Summary

Constructors

Constructor and Description
PythonStreamer(AbstractRichFunction function, int id, boolean usesByteArray)

Method Summary

Modifier and Type	Method and Description
void	close() Closes this streamer.
void	open() Starts the python script.
void	sendBroadCastVariables(Configuration config) Sends all broadcast-variables encoded in the configuration to the external process.
void	streamBufferWithGroups(Iterator i1, Iterator i2, Collector c) Sends all values contained in both iterators to the external process and collects all results.
void	streamBufferWithoutGroups(Iterator i, Collector c) Sends all values contained in the iterator to the external process and collects all results.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

protected static final org.slf4j.Logger LOG

server

protected ServerSocket server

socket

protected Socket socket

in

protected DataInputStream in

out

protected DataOutputStream out

port

protected int port

sender

protected PythonSender sender

receiver

protected PythonReceiver receiver

msg

protected StringBuilder msg

function

protected final AbstractRichFunction function

Constructor Detail

PythonStreamer

public PythonStreamer(AbstractRichFunction function,
                      int id,
                      boolean usesByteArray)

Method Detail

open

public void open()
          throws IOException

Starts the python script.

Throws:

IOException

close

public void close()
           throws IOException

Closes this streamer.

Throws:

IOException

sendBroadCastVariables

public final void sendBroadCastVariables(Configuration config)
                                  throws IOException

Sends all broadcast-variables encoded in the configuration to the external process.

Parameters:

config - configuration object containing broadcast-variable count and names

Throws:

IOException

streamBufferWithoutGroups

public final void streamBufferWithoutGroups(Iterator i,
                                            Collector c)
                                     throws IOException

Sends all values contained in the iterator to the external process and collects all results.

Parameters:

i - iterator

c - collector

Throws:

IOException

streamBufferWithGroups

public final void streamBufferWithGroups(Iterator i1,
                                         Iterator i2,
                                         Collector c)
                                  throws IOException

Sends all values contained in both iterators to the external process and collects all results.

Parameters:

i1 - iterator

i2 - iterator

c - collector

Throws:

IOException