IN - Type of the elements in the input of the sink that are also the elements to be
written to its output@Experimental public class FileSink<IN> extends Object implements org.apache.flink.api.connector.sink2.StatefulSink<IN,FileWriterBucketState>, org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink<IN,FileSinkCommittable>, org.apache.flink.api.connector.sink2.StatefulSink.WithCompatibleState, org.apache.flink.streaming.api.connector.sink2.WithPreCommitTopology<IN,FileSinkCommittable>
FileSystem files within buckets. This
sink achieves exactly-once semantics for both BATCH and STREAMING.
When creating the sink a basePath must be specified. The base directory contains one
directory for every bucket. The bucket directories themselves contain several part files, with at
least one for each parallel subtask of the sink which is writing data to that bucket. These part
files contain the actual output data.
The sink uses a BucketAssigner to determine in which bucket directory each element
should be written to inside the base directory. The BucketAssigner can, for example, roll
on every checkpoint or use time or a property of the element to determine the bucket directory.
The default BucketAssigner is a DateTimeBucketAssigner which will create one new
bucket every hour. You can specify a custom BucketAssigner using the setBucketAssigner(bucketAssigner) method, after calling forRowFormat(Path,
Encoder) or forBulkFormat(Path, BulkWriter.Factory).
The names of the part files could be defined using OutputFileConfig. This
configuration contains a part prefix and a part suffix that will be used with a random uid
assigned to each subtask of the sink and a rolling counter to determine the file names. For
example with a prefix "prefix" and a suffix ".ext", a file named "prefix-81fc4980-a6af-41c8-9937-9939408a734b-17.ext" contains the data from subtask with uid
81fc4980-a6af-41c8-9937-9939408a734b of the sink and is the 17th part-file
created by that subtask.
Part files roll based on the user-specified RollingPolicy. By default, a DefaultRollingPolicy is used for row-encoded sink output; a OnCheckpointRollingPolicy is
used for bulk-encoded sink output.
In some scenarios, the open buckets are required to change based on time. In these cases, the
user can specify a bucketCheckInterval (by default 1m) and the sink will check
periodically and roll the part file if the specified rolling policy says so.
Part files can be in one of three states: in-progress, pending or finished. The reason for this is how the sink works to provide exactly-once semantics and
fault-tolerance. The part file that is currently being written to is in-progress. Once a
part file is closed for writing it becomes pending. When a checkpoint is successful (for
STREAMING) or at the end of the job (for BATCH) the currently pending files will
be moved to finished.
For STREAMING in order to guarantee exactly-once semantics in case of a failure, the
sink should roll back to the state it had when that last successful checkpoint occurred. To this
end, when restoring, the restored files in pending state are transferred into the finished state while any in-progress files are rolled back, so that they do not contain
data that arrived after the checkpoint from which we restore.
FileSink also support compacting small files to accelerate the access speed of the resulted
files. Compaction could be enabled via enableCompact. Once enabled, the compaction could
only be disabled via calling disableCompact explicitly, otherwise there might be data
loss.
| 限定符和类型 | 类和说明 |
|---|---|
static class |
FileSink.BulkFormatBuilder<IN,T extends FileSink.BulkFormatBuilder<IN,T>>
A builder for configuring the sink for bulk-encoding formats, e.g.
|
static class |
FileSink.DefaultBulkFormatBuilder<IN>
Builder for the vanilla
FileSink using a bulk format. |
static class |
FileSink.DefaultRowFormatBuilder<IN>
Builder for the vanilla
FileSink using a row format. |
static class |
FileSink.RowFormatBuilder<IN,T extends FileSink.RowFormatBuilder<IN,T>>
A builder for configuring the sink for row-wise encoding formats.
|
org.apache.flink.api.connector.sink2.StatefulSink.StatefulSinkWriter<InputT,WriterStateT>, org.apache.flink.api.connector.sink2.StatefulSink.WithCompatibleState| 限定符和类型 | 方法和说明 |
|---|---|
org.apache.flink.streaming.api.datastream.DataStream<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>> |
addPreCommitTopology(org.apache.flink.streaming.api.datastream.DataStream<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>> committableStream) |
org.apache.flink.api.connector.sink2.Committer<FileSinkCommittable> |
createCommitter() |
FileWriter<IN> |
createWriter(org.apache.flink.api.connector.sink2.Sink.InitContext context) |
static <IN> FileSink.DefaultBulkFormatBuilder<IN> |
forBulkFormat(org.apache.flink.core.fs.Path basePath,
org.apache.flink.api.common.serialization.BulkWriter.Factory<IN> bulkWriterFactory) |
static <IN> FileSink.DefaultRowFormatBuilder<IN> |
forRowFormat(org.apache.flink.core.fs.Path basePath,
org.apache.flink.api.common.serialization.Encoder<IN> encoder) |
org.apache.flink.core.io.SimpleVersionedSerializer<FileSinkCommittable> |
getCommittableSerializer() |
Collection<String> |
getCompatibleWriterStateNames() |
org.apache.flink.core.io.SimpleVersionedSerializer<FileWriterBucketState> |
getWriterStateSerializer() |
org.apache.flink.api.connector.sink2.StatefulSink.StatefulSinkWriter<IN,FileWriterBucketState> |
restoreWriter(org.apache.flink.api.connector.sink2.Sink.InitContext context,
Collection<FileWriterBucketState> recoveredState) |
public FileWriter<IN> createWriter(org.apache.flink.api.connector.sink2.Sink.InitContext context) throws IOException
createWriter 在接口中 org.apache.flink.api.connector.sink2.Sink<IN>createWriter 在接口中 org.apache.flink.api.connector.sink2.StatefulSink<IN,FileWriterBucketState>createWriter 在接口中 org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink<IN,FileSinkCommittable>IOExceptionpublic org.apache.flink.api.connector.sink2.StatefulSink.StatefulSinkWriter<IN,FileWriterBucketState> restoreWriter(org.apache.flink.api.connector.sink2.Sink.InitContext context, Collection<FileWriterBucketState> recoveredState) throws IOException
restoreWriter 在接口中 org.apache.flink.api.connector.sink2.StatefulSink<IN,FileWriterBucketState>IOExceptionpublic org.apache.flink.core.io.SimpleVersionedSerializer<FileWriterBucketState> getWriterStateSerializer()
getWriterStateSerializer 在接口中 org.apache.flink.api.connector.sink2.StatefulSink<IN,FileWriterBucketState>public org.apache.flink.api.connector.sink2.Committer<FileSinkCommittable> createCommitter() throws IOException
createCommitter 在接口中 org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink<IN,FileSinkCommittable>IOExceptionpublic org.apache.flink.core.io.SimpleVersionedSerializer<FileSinkCommittable> getCommittableSerializer()
getCommittableSerializer 在接口中 org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink<IN,FileSinkCommittable>public Collection<String> getCompatibleWriterStateNames()
getCompatibleWriterStateNames 在接口中 org.apache.flink.api.connector.sink2.StatefulSink.WithCompatibleStatepublic static <IN> FileSink.DefaultRowFormatBuilder<IN> forRowFormat(org.apache.flink.core.fs.Path basePath, org.apache.flink.api.common.serialization.Encoder<IN> encoder)
public static <IN> FileSink.DefaultBulkFormatBuilder<IN> forBulkFormat(org.apache.flink.core.fs.Path basePath, org.apache.flink.api.common.serialization.BulkWriter.Factory<IN> bulkWriterFactory)
public org.apache.flink.streaming.api.datastream.DataStream<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>> addPreCommitTopology(org.apache.flink.streaming.api.datastream.DataStream<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>> committableStream)
addPreCommitTopology 在接口中 org.apache.flink.streaming.api.connector.sink2.WithPreCommitTopology<IN,FileSinkCommittable>Copyright © 2014–2022 The Apache Software Foundation. All rights reserved.