org.apache.flink.connectors.hive

Class HiveTableSink

java.lang.Object

org.apache.flink.connectors.hive.HiveTableSink

All Implemented Interfaces:

SupportsOverwrite, SupportsPartitioning, DynamicTableSink

public class HiveTableSink
extends Object
implements DynamicTableSink, SupportsPartitioning, SupportsOverwrite

Table sink to write to Hive tables.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.table.connector.sink.DynamicTableSink

DynamicTableSink.Context, DynamicTableSink.DataStructureConverter, DynamicTableSink.SinkRuntimeProvider

Field Summary

Fields

Modifier and Type	Field and Description
static String	BATCH_COMPACT_WRITER_OP_NAME

Constructor Summary

Constructors

Constructor and Description
HiveTableSink(ReadableConfig flinkConf, org.apache.hadoop.mapred.JobConf jobConf, ObjectIdentifier identifier, ResolvedCatalogTable table, Integer configuredSinkParallelism)

Method Summary

Modifier and Type	Method and Description
void	applyOverwrite(boolean overwrite) Provides whether existing data should be overwritten or not.
void	applyStaticPartition(Map<String,String> partition) Provides the static part of a partition.
String	asSummaryString() Returns a string that summarizes this sink for printing to a console or log.
DynamicTableSink	copy() Creates a copy of this instance during planning.
ChangelogMode	getChangelogMode(ChangelogMode requestedMode) Returns the set of changes that the sink accepts during runtime.
org.apache.hadoop.mapred.JobConf	getJobConf()
DynamicTableSink.SinkRuntimeProvider	getSinkRuntimeProvider(DynamicTableSink.Context context) Returns a provider of runtime implementation for writing the data.
boolean	requiresPartitionGrouping(boolean supportsGrouping) Returns whether data needs to be grouped by partition before it is consumed by the sink.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

BATCH_COMPACT_WRITER_OP_NAME

public static final String BATCH_COMPACT_WRITER_OP_NAME

See Also:

Constant Field Values

Constructor Detail

HiveTableSink

public HiveTableSink(ReadableConfig flinkConf,
                     org.apache.hadoop.mapred.JobConf jobConf,
                     ObjectIdentifier identifier,
                     ResolvedCatalogTable table,
                     @Nullable
                     Integer configuredSinkParallelism)

Method Detail

getSinkRuntimeProvider

public DynamicTableSink.SinkRuntimeProvider getSinkRuntimeProvider(DynamicTableSink.Context context)

Description copied from interface: DynamicTableSink

Returns a provider of runtime implementation for writing the data.

There might exist different interfaces for runtime implementation which is why DynamicTableSink.SinkRuntimeProvider serves as the base interface. Concrete DynamicTableSink.SinkRuntimeProvider interfaces might be located in other Flink modules.

Independent of the provider interface, the table runtime expects that a sink implementation accepts internal data structures (see RowData for more information).

The given DynamicTableSink.Context offers utilities by the planner for creating runtime implementation with minimal dependencies to internal data structures.

SinkProvider is the recommended core interface. SinkFunctionProvider in flink-table-api-java-bridge and OutputFormatProvider are available for backwards compatibility.

Specified by:

getSinkRuntimeProvider in interface DynamicTableSink

See Also:

SinkProvider

requiresPartitionGrouping

public boolean requiresPartitionGrouping(boolean supportsGrouping)

Description copied from interface: SupportsPartitioning

Returns whether data needs to be grouped by partition before it is consumed by the sink. By default, this is not required from the runtime and records arrive in arbitrary partition order.

If this method returns true, the sink can expect that all records will be grouped by the partition keys before consumed by the sink. In other words: The sink will receive all elements of one partition and then all elements of another partition. Elements of different partitions will not be mixed. For some sinks, this can be used to reduce the number of partition writers and improve writing performance by writing one partition at a time.

The given argument indicates whether the current execution mode supports grouping or not. For example, depending on the execution mode a sorting operation might not be available during runtime.

Specified by:

requiresPartitionGrouping in interface SupportsPartitioning

Parameters:

supportsGrouping - whether the current execution mode supports grouping

Returns:

whether data need to be grouped by partition before consumed by the sink. If supportsGrouping is false, it should never return true, otherwise the planner will fail.

applyStaticPartition

public void applyStaticPartition(Map<String,String> partition)

Description copied from interface: SupportsPartitioning

Provides the static part of a partition.

A single partition maps each partition key to a partition value. Depending on the user-defined statement, the partition might not include all partition keys.

See the documentation of SupportsPartitioning for more information.

Specified by:

applyStaticPartition in interface SupportsPartitioning

Parameters:

partition - user-defined (possibly partial) static partition

applyOverwrite

public void applyOverwrite(boolean overwrite)

Description copied from interface: SupportsOverwrite

Provides whether existing data should be overwritten or not.

Specified by:

applyOverwrite in interface SupportsOverwrite

getChangelogMode

public ChangelogMode getChangelogMode(ChangelogMode requestedMode)

Description copied from interface: DynamicTableSink

Returns the set of changes that the sink accepts during runtime.

The planner can make suggestions but the sink has the final decision what it requires. If the planner does not support this mode, it will throw an error. For example, the sink can return that it only supports ChangelogMode.insertOnly().

Specified by:

getChangelogMode in interface DynamicTableSink

Parameters:

requestedMode - expected set of changes by the current plan

copy

public DynamicTableSink copy()

Description copied from interface: DynamicTableSink

Creates a copy of this instance during planning. The copy should be a deep copy of all mutable members.

Specified by:

copy in interface DynamicTableSink

asSummaryString

public String asSummaryString()

Description copied from interface: DynamicTableSink

Returns a string that summarizes this sink for printing to a console or log.

Specified by:

asSummaryString in interface DynamicTableSink

getJobConf

@VisibleForTesting
public org.apache.hadoop.mapred.JobConf getJobConf()