@Deprecated public class AvroKeyValueSinkWriter<K,V> extends StreamWriterBase<Tuple2<K,V>> implements Writer<Tuple2<K,V>>, InputTypeConfigurable
Usage:
BucketingSink<Tuple2<Long, Long>> sink = new BucketingSink<Tuple2<Long, Long>>("/tmp/path");
sink.setBucketer(new DateTimeBucketer<Tuple2<Long, Long>>("yyyy-MM-dd/HH/mm/"));
sink.setPendingSuffix(".avro");
Map<String, String> properties = new HashMap<>();
Schema longSchema = Schema.create(Type.LONG);
String keySchema = longSchema.toString();
String valueSchema = longSchema.toString();
properties.put(AvroKeyValueSinkWriter.CONF_OUTPUT_KEY_SCHEMA, keySchema);
properties.put(AvroKeyValueSinkWriter.CONF_OUTPUT_VALUE_SCHEMA, valueSchema);
properties.put(AvroKeyValueSinkWriter.CONF_COMPRESS, Boolean.toString(true));
properties.put(AvroKeyValueSinkWriter.CONF_COMPRESS_CODEC, DataFileConstants.SNAPPY_CODEC);
sink.setWriter(new AvroKeyValueSinkWriter<Long, Long>(properties));
sink.setBatchSize(1024 * 1024 * 64); // this is 64 MB,
Modifier and Type | Class and Description |
---|---|
static class |
AvroKeyValueSinkWriter.AvroKeyValue<K,V>
Deprecated.
A reusable Avro generic record for writing key/value pairs to the
file.
|
Modifier and Type | Field and Description |
---|---|
static String |
CONF_COMPRESS
Deprecated.
|
static String |
CONF_COMPRESS_CODEC
Deprecated.
|
static String |
CONF_DEFLATE_LEVEL
Deprecated.
|
static String |
CONF_OUTPUT_KEY_SCHEMA
Deprecated.
|
static String |
CONF_OUTPUT_VALUE_SCHEMA
Deprecated.
|
static String |
CONF_XZ_LEVEL
Deprecated.
|
Modifier | Constructor and Description |
---|---|
protected |
AvroKeyValueSinkWriter(AvroKeyValueSinkWriter<K,V> other)
Deprecated.
|
|
AvroKeyValueSinkWriter(Map<String,String> properties)
Deprecated.
C'tor for the writer.
|
Modifier and Type | Method and Description |
---|---|
void |
close()
Deprecated.
Closes the
Writer . |
AvroKeyValueSinkWriter<K,V> |
duplicate()
Deprecated.
Duplicates the
Writer . |
long |
flush()
Deprecated.
Flushes out any internally held data, and returns the offset that the file
must be truncated to at recovery.
|
void |
open(org.apache.hadoop.fs.FileSystem fs,
org.apache.hadoop.fs.Path path)
Deprecated.
Initializes the
Writer for a newly opened bucket file. |
void |
setInputType(TypeInformation<?> type,
ExecutionConfig executionConfig)
Deprecated.
Method that is called on an
OutputFormat when it is passed to
the DataSet's output method. |
void |
write(Tuple2<K,V> element)
Deprecated.
Writes one element to the bucket file.
|
getPos, getStream, isSyncOnFlush, setSyncOnFlush
public static final String CONF_OUTPUT_KEY_SCHEMA
public static final String CONF_OUTPUT_VALUE_SCHEMA
public static final String CONF_COMPRESS
public static final String CONF_COMPRESS_CODEC
public static final String CONF_DEFLATE_LEVEL
public static final String CONF_XZ_LEVEL
public AvroKeyValueSinkWriter(Map<String,String> properties)
You can provide different properties that will be used to configure avro key-value writer as simple properties map(see example above)
properties
- protected AvroKeyValueSinkWriter(AvroKeyValueSinkWriter<K,V> other)
public void open(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) throws IOException
Writer
Writer
for a newly opened bucket file.
Any internal per-bucket initialization should be performed here.public void close() throws IOException
Writer
Writer
. If the writer is already closed, no action will be
taken. The call should close all state related to the current output file,
including the output stream opened in open
.public long flush() throws IOException
Writer
public void write(Tuple2<K,V> element) throws IOException
Writer
public void setInputType(TypeInformation<?> type, ExecutionConfig executionConfig)
InputTypeConfigurable
OutputFormat
when it is passed to
the DataSet's output method. May be used to configures the output format based on the data type.setInputType
in interface InputTypeConfigurable
type
- The data type of the input.executionConfig
- The execution config for this parallel execution.Copyright © 2014–2020 The Apache Software Foundation. All rights reserved.