org.apache.flink.streaming.api.functions.sink.filesystem

Class StreamingFileSink.BulkFormatBuilder<IN,BucketID,T extends StreamingFileSink.BulkFormatBuilder<IN,BucketID,T>>

java.lang.Object

org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.BucketsBuilder<IN,BucketID,T>

org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.BulkFormatBuilder<IN,BucketID,T>

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

StreamingFileSink.DefaultBulkFormatBuilder

Enclosing class:

StreamingFileSink<IN>

@PublicEvolving
public static class StreamingFileSink.BulkFormatBuilder<IN,BucketID,T extends StreamingFileSink.BulkFormatBuilder<IN,BucketID,T>>
extends StreamingFileSink.BucketsBuilder<IN,BucketID,T>

A builder for configuring the sink for bulk-encoding formats, e.g. Parquet/ORC.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.BucketsBuilder

DEFAULT_BUCKET_CHECK_INTERVAL

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	BulkFormatBuilder(org.apache.flink.core.fs.Path basePath, org.apache.flink.api.common.serialization.BulkWriter.Factory<IN> writerFactory, org.apache.flink.streaming.api.functions.sink.filesystem.BucketAssigner<IN,BucketID> assigner)
protected	BulkFormatBuilder(org.apache.flink.core.fs.Path basePath, org.apache.flink.api.common.serialization.BulkWriter.Factory<IN> writerFactory, org.apache.flink.streaming.api.functions.sink.filesystem.BucketAssigner<IN,BucketID> assigner, org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.CheckpointRollingPolicy<IN,BucketID> policy, long bucketCheckInterval, BucketFactory<IN,BucketID> bucketFactory, org.apache.flink.streaming.api.functions.sink.filesystem.OutputFileConfig outputFileConfig)

Method Summary

Modifier and Type	Method and Description
StreamingFileSink<IN>	build() Creates the actual sink.
Buckets<IN,BucketID>	createBuckets(int subtaskIndex)
org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter<IN,BucketID>	createBucketWriter()
long	getBucketCheckInterval()
T	withBucketAssigner(org.apache.flink.streaming.api.functions.sink.filesystem.BucketAssigner<IN,BucketID> assigner)
T	withBucketCheckInterval(long interval)
<ID> StreamingFileSink.BulkFormatBuilder<IN,ID,? extends StreamingFileSink.BulkFormatBuilder<IN,ID,?>>	withNewBucketAssigner(org.apache.flink.streaming.api.functions.sink.filesystem.BucketAssigner<IN,ID> assigner)
T	withOutputFileConfig(org.apache.flink.streaming.api.functions.sink.filesystem.OutputFileConfig outputFileConfig)
T	withRollingPolicy(org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.CheckpointRollingPolicy<IN,BucketID> rollingPolicy)

Methods inherited from class org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.BucketsBuilder

self

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BulkFormatBuilder

protected BulkFormatBuilder(org.apache.flink.core.fs.Path basePath,
                            org.apache.flink.api.common.serialization.BulkWriter.Factory<IN> writerFactory,
                            org.apache.flink.streaming.api.functions.sink.filesystem.BucketAssigner<IN,BucketID> assigner)

BulkFormatBuilder

protected BulkFormatBuilder(org.apache.flink.core.fs.Path basePath,
                            org.apache.flink.api.common.serialization.BulkWriter.Factory<IN> writerFactory,
                            org.apache.flink.streaming.api.functions.sink.filesystem.BucketAssigner<IN,BucketID> assigner,
                            org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.CheckpointRollingPolicy<IN,BucketID> policy,
                            long bucketCheckInterval,
                            BucketFactory<IN,BucketID> bucketFactory,
                            org.apache.flink.streaming.api.functions.sink.filesystem.OutputFileConfig outputFileConfig)

Method Detail

getBucketCheckInterval

public long getBucketCheckInterval()

withBucketCheckInterval

public T withBucketCheckInterval(long interval)

withBucketAssigner

public T withBucketAssigner(org.apache.flink.streaming.api.functions.sink.filesystem.BucketAssigner<IN,BucketID> assigner)

withRollingPolicy

public T withRollingPolicy(org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.CheckpointRollingPolicy<IN,BucketID> rollingPolicy)

withOutputFileConfig

public T withOutputFileConfig(org.apache.flink.streaming.api.functions.sink.filesystem.OutputFileConfig outputFileConfig)

withNewBucketAssigner

public <ID> StreamingFileSink.BulkFormatBuilder<IN,ID,? extends StreamingFileSink.BulkFormatBuilder<IN,ID,?>> withNewBucketAssigner(org.apache.flink.streaming.api.functions.sink.filesystem.BucketAssigner<IN,ID> assigner)

build

public StreamingFileSink<IN> build()

Creates the actual sink.

createBucketWriter

@Internal
public org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter<IN,BucketID> createBucketWriter()
                                                                                                                throws IOException

Specified by:

createBucketWriter in class StreamingFileSink.BucketsBuilder<IN,BucketID,T extends StreamingFileSink.BulkFormatBuilder<IN,BucketID,T>>

Throws:

IOException

createBuckets

@Internal
public Buckets<IN,BucketID> createBuckets(int subtaskIndex)
                                             throws IOException

Specified by:

createBuckets in class StreamingFileSink.BucketsBuilder<IN,BucketID,T extends StreamingFileSink.BulkFormatBuilder<IN,BucketID,T>>

Throws:

IOException