@Internal @Deprecated public class BufferSpiller extends AbstractBufferStorage
BufferSpiller
takes the buffers and events from a data stream and adds them to a spill file.
After a number of elements have been spilled, the spiller can "roll over": It presents the spilled
elements as a readable sequence, and opens a new spill file.
This implementation buffers data effectively in the OS cache, which gracefully extends to the disk. Most data is written and re-read milliseconds later. The file is deleted after the read. Consequently, in most cases, the data will never actually hit the physical disks.
IMPORTANT: The SpilledBufferOrEventSequences created by this spiller all reuse the same reading memory (to reduce overhead) and can consequently not be read concurrently.
Modifier and Type | Class and Description |
---|---|
static class |
BufferSpiller.SpilledBufferOrEventSequence
Deprecated.
|
currentBuffered, LOG, maxBufferedBytes, queuedBuffered, rolledBytes, taskName
Constructor and Description |
---|
BufferSpiller(IOManager ioManager,
int pageSize)
Deprecated.
|
BufferSpiller(IOManager ioManager,
int pageSize,
long maxBufferedBytes)
Deprecated.
|
BufferSpiller(IOManager ioManager,
int pageSize,
long maxBufferedBytes,
String taskName)
Deprecated.
Creates a new
BufferSpiller , spilling to one of the I/O manager's temp directories. |
Modifier and Type | Method and Description |
---|---|
void |
add(BufferOrEvent boe)
Deprecated.
Adds a buffer or event to the
BufferStorage . |
void |
close()
Deprecated.
Cleans up the current spilling channel and file.
|
long |
getPendingBytes()
Deprecated.
Gets the number of bytes written in the current spill file.
|
BufferOrEventSequence |
rollOverReusingResources()
Deprecated.
NOTE: The BufferOrEventSequences created by this method all reuse the same reading memory
(to reduce overhead) and can consequently not be read concurrently with each other.
|
BufferOrEventSequence |
rollOverWithoutReusingResources()
Deprecated.
The BufferOrEventSequence returned by this method is safe for concurrent consumption with
any previously returned sequence.
|
completeBufferedSequence, getMaxBufferedBytes, getRolledBytes, isEmpty, isFull, pollNext, rollOver
public BufferSpiller(IOManager ioManager, int pageSize) throws IOException
IOException
public BufferSpiller(IOManager ioManager, int pageSize, long maxBufferedBytes) throws IOException
IOException
public BufferSpiller(IOManager ioManager, int pageSize, long maxBufferedBytes, String taskName) throws IOException
BufferSpiller
, spilling to one of the I/O manager's temp directories.ioManager
- The I/O manager for access to the temp directories.pageSize
- The page size used to re-create spilled buffers.maxBufferedBytes
- The maximum bytes to be buffered before the checkpoint aborts.taskName
- The task name for logging.IOException
- Thrown if the temp files for spilling cannot be initialized.public void add(BufferOrEvent boe) throws IOException
BufferStorage
BufferStorage
.boe
- The buffer or event to be added into the blocker.IOException
public BufferOrEventSequence rollOverReusingResources() throws IOException
To create a sequence that can be read concurrently with the previous BufferOrEventSequence,
use the rollOverWithoutReusingResources()
()} method.
rollOverReusingResources
in class AbstractBufferStorage
IOException
- Thrown, if the readable sequence could not be created, or no new spill
file could be created.public BufferOrEventSequence rollOverWithoutReusingResources() throws IOException
rollOverWithoutReusingResources
in class AbstractBufferStorage
IOException
- Thrown, if the readable sequence could not be created, or no new spill
file could be created.public void close() throws IOException
Does not clean up the SpilledBufferOrEventSequences generated by calls to
rollOver(boolean false)
.
close
in interface AutoCloseable
close
in interface BufferStorage
close
in class AbstractBufferStorage
IOException
- Thrown if channel closing or file deletion fail.public long getPendingBytes()
Copyright © 2014–2020 The Apache Software Foundation. All rights reserved.