org.apache.flink.connectors.hive.util

Class HivePartitionUtils

java.lang.Object

org.apache.flink.connectors.hive.util.HivePartitionUtils

public class HivePartitionUtils
extends Object

Utils to load hive partitions from HiveMetaStore.

Constructor Summary

Constructors

Constructor and Description
HivePartitionUtils()

Method Summary

Modifier and Type	Method and Description
static CatalogPartitionSpec	createPartitionSpec(String hivePartitionName, String defaultPartitionName) Creates a CatalogPartitionSpec from a Hive partition name string.
static List<HiveTablePartition>	deserializeHiveTablePartition(List<byte[]> partitionBytes)
static List<HiveTablePartition>	getAllPartitions(org.apache.hadoop.mapred.JobConf jobConf, String hiveVersion, ObjectPath tablePath, List<String> partitionColNames, List<Map<String,String>> remainingPartitions) Returns all HiveTablePartitions of a hive table, returns single HiveTablePartition if the hive table is not partitioned.
static org.apache.hadoop.fs.FileStatus[]	getFileStatusRecurse(org.apache.hadoop.fs.Path path, int expectLevel, org.apache.hadoop.fs.FileSystem fs)
static List<String>	getPartitionNames(List<Map<String,String>> partitionsSpec, List<String> partitionColNames, String defaultStr) Get the partitions' name by partitions' spec.
static Map<String,Object>	parsePartitionValues(Map<String,String> partitionSpecs, String[] fieldNames, DataType[] fieldTypes, String defaultPartitionName, HiveShim shim) Parse partition string specs into object values.
static List<String>	partitionSpecToValues(Map<String,String> spec, List<String> partitionColNames)
static Object	restorePartitionValueFromType(HiveShim shim, String valStr, LogicalType partitionType, String defaultPartitionName)
static List<byte[]>	serializeHiveTablePartition(List<HiveTablePartition> hiveTablePartitions)
static HiveTablePartition	toHiveTablePartition(List<String> partitionKeys, Properties tableProps, org.apache.hadoop.hive.metastore.api.Partition partition)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HivePartitionUtils

public HivePartitionUtils()

Method Detail

parsePartitionValues

public static Map<String,Object> parsePartitionValues(Map<String,String> partitionSpecs,
                                                      String[] fieldNames,
                                                      DataType[] fieldTypes,
                                                      String defaultPartitionName,
                                                      HiveShim shim)

Parse partition string specs into object values.

restorePartitionValueFromType

public static Object restorePartitionValueFromType(HiveShim shim,
                                                   String valStr,
                                                   LogicalType partitionType,
                                                   String defaultPartitionName)

getAllPartitions

public static List<HiveTablePartition> getAllPartitions(org.apache.hadoop.mapred.JobConf jobConf,
                                                        String hiveVersion,
                                                        ObjectPath tablePath,
                                                        List<String> partitionColNames,
                                                        List<Map<String,String>> remainingPartitions)

Returns all HiveTablePartitions of a hive table, returns single HiveTablePartition if the hive table is not partitioned.

getPartitionNames

public static List<String> getPartitionNames(List<Map<String,String>> partitionsSpec,
                                             List<String> partitionColNames,
                                             String defaultStr)

Get the partitions' name by partitions' spec.

Parameters:

partitionsSpec - a list contains the spec of the partitions, one of which is for one partition. The map for the spec of partition can be unordered.

partitionColNames - the partition column's name

defaultStr - the default value used to make partition name when the key or value for the partition's spec partition column in the spec is null or empty string.

Returns:

a list contains the partitions' name like "p1=v1/p2=v2", one of which is for one partition.

createPartitionSpec

public static CatalogPartitionSpec createPartitionSpec(String hivePartitionName,
                                                       String defaultPartitionName)

Creates a CatalogPartitionSpec from a Hive partition name string. Example of Hive partition name string - "name=bob/year=2019". If the partition name for the given partition column is equal to , the partition value in returned CatalogPartitionSpec will be null.

partitionSpecToValues

public static List<String> partitionSpecToValues(Map<String,String> spec,
                                                 List<String> partitionColNames)

toHiveTablePartition

public static HiveTablePartition toHiveTablePartition(List<String> partitionKeys,
                                                      Properties tableProps,
                                                      org.apache.hadoop.hive.metastore.api.Partition partition)

getFileStatusRecurse

public static org.apache.hadoop.fs.FileStatus[] getFileStatusRecurse(org.apache.hadoop.fs.Path path,
                                                                     int expectLevel,
                                                                     org.apache.hadoop.fs.FileSystem fs)

serializeHiveTablePartition

public static List<byte[]> serializeHiveTablePartition(List<HiveTablePartition> hiveTablePartitions)

deserializeHiveTablePartition

public static List<HiveTablePartition> deserializeHiveTablePartition(List<byte[]> partitionBytes)