org.apache.flink.table.runtime.operators.aggregate

Class BytesHashMap

java.lang.Object

org.apache.flink.table.runtime.operators.aggregate.BytesHashMap

public class BytesHashMap
extends Object

Bytes based hash map. It can be used for performing aggregations where the aggregated values are fixed-width, because the data is stored in continuous memory, AggBuffer of variable length cannot be applied to this HashMap. The KeyValue form in hash map is designed to reduce the cost of key fetching in lookup. The memory is divided into two areas:

- Bucket area: this contains: pointer + hashcode. Bytes 0 to 8: a pointer to the record in the record area Bytes 8 to 16: a holds key's full 32-bit hashcode

- Record area: this contains the actual data in linked list records. A BytesHashMap's record has four parts: Bytes 0 to 4: len(k) Bytes 4 to 4 + len(k): key data Bytes 4 + len(k) to 8 + len(k): len(v) Bytes 8 + len(k) to 8 + len(k) + len(v): value data

BytesHashMap are influenced by Apache Spark BytesToBytesMap.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	BytesHashMap.Entry BytesHashMap Entry contains key and value field.
static class	BytesHashMap.LookupInfo Handle returned by lookup(BinaryRowData) function.

Field Summary

Fields

Modifier and Type	Field and Description
static int	BUCKET_SIZE
static int	RECORD_EXTRA_LENGTH

Constructor Summary

Constructors

Constructor and Description
BytesHashMap(Object owner, MemoryManager memoryManager, long memorySize, LogicalType[] keyTypes, LogicalType[] valueTypes)
BytesHashMap(Object owner, MemoryManager memoryManager, long memorySize, LogicalType[] keyTypes, LogicalType[] valueTypes, boolean inferBucketMemory)

Method Summary

Modifier and Type	Method and Description
BinaryRowData	append(BytesHashMap.LookupInfo info, BinaryRowData value) Append an value into the hash map's record area.
void	free() release the map's record and bucket area's memory segments.
void	free(boolean reservedRecordMemory)
List<MemorySegment>	getBucketAreaMemorySegments()
MutableObjectIterator<BytesHashMap.Entry>	getEntryIterator() Returns a destructive iterator for iterating over the entries of this map.
long	getNumElements()
long	getNumSpillFiles()
ArrayList<MemorySegment>	getRecordAreaMemorySegments()
long	getSpillInBytes()
long	getUsedMemoryInBytes()
BytesHashMap.LookupInfo	lookup(BinaryRowData key)
void	reset() reset the map's record and bucket area's memory segments for reusing.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

BUCKET_SIZE

public static final int BUCKET_SIZE

See Also:

Constant Field Values

RECORD_EXTRA_LENGTH

public static final int RECORD_EXTRA_LENGTH

See Also:

Constant Field Values

Constructor Detail

BytesHashMap

public BytesHashMap(Object owner,
                    MemoryManager memoryManager,
                    long memorySize,
                    LogicalType[] keyTypes,
                    LogicalType[] valueTypes)

BytesHashMap

public BytesHashMap(Object owner,
                    MemoryManager memoryManager,
                    long memorySize,
                    LogicalType[] keyTypes,
                    LogicalType[] valueTypes,
                    boolean inferBucketMemory)

Method Detail

lookup

public BytesHashMap.LookupInfo lookup(BinaryRowData key)

Parameters:

key - by which looking up the value in the hash map. Only support the key in the BinaryRowData form who has only one MemorySegment.

Returns:

BytesHashMap.LookupInfo

append

public BinaryRowData append(BytesHashMap.LookupInfo info,
                            BinaryRowData value)
                     throws IOException

Append an value into the hash map's record area.

Returns:

An BinaryRowData mapping to the memory segments in the map's record area belonging to the newly appended value.

Throws:

EOFException - if the map can't allocate much more memory.

IOException

getNumSpillFiles

public long getNumSpillFiles()

getUsedMemoryInBytes

public long getUsedMemoryInBytes()

getSpillInBytes

public long getSpillInBytes()

getNumElements

public long getNumElements()

getEntryIterator

public MutableObjectIterator<BytesHashMap.Entry> getEntryIterator()

Returns a destructive iterator for iterating over the entries of this map. It frees each page as it moves onto next one. Notice: it is illegal to call any method on the map after `destructiveIterator()` has been called.

Returns:

an entry iterator for iterating the value appended in the hash map.

getRecordAreaMemorySegments

public ArrayList<MemorySegment> getRecordAreaMemorySegments()

Returns:

the underlying memory segments of the hash map's record area

getBucketAreaMemorySegments

public List<MemorySegment> getBucketAreaMemorySegments()

free

public void free()

release the map's record and bucket area's memory segments.

free

public void free(boolean reservedRecordMemory)

Parameters:

reservedRecordMemory - reserved fixed memory or not.

reset

public void reset()

reset the map's record and bucket area's memory segments for reusing.