Type Members

1.  class FileStreamSink extends Sink with Logging

   A sink that writes out results to parquet files.

   A sink that writes out results to parquet files. Each batch is written out to a unique directory. After all of the files in a batch have been successfully written, the list of file paths is appended to the log atomically. In the case of partial failures, some duplicate data may be present in the target directory, but only one copy of each file will be present in the log.

2.  class FileStreamSinkLog extends CompactibleFileStreamLog[SinkFileStatus]

   A special log for FileStreamSink.

   A special log for FileStreamSink. It will write one log file for each batch. The first line of the log file is the version number, and there are multiple JSON lines following. Each JSON line is a JSON format of SinkFileStatus.

   As reading from many small files is usually pretty slow, FileStreamSinkLog will compact log files every "spark.sql.sink.file.log.compactLen" batches into a big file. When doing a compaction, it will read all old log files and merge them with the new batch. During the compaction, it will also delete the files that are deleted (marked by SinkFileStatus.action). When the reader uses allFiles to list all files, this method only returns the visible files (drops the deleted files).

3.  case class SinkFileStatus(path: String, size: Long, isDir: Boolean, modificationTime: Long, blockReplication: Int, blockSize: Long, action: String) extends Product with Serializable

   The status of a file outputted by FileStreamSink.

   The status of a file outputted by FileStreamSink. A file is visible only if it appears in the sink log and its action is not "delete".

   path

   the file path as a uri-encoded string.

   size

   the file size.

   isDir

   whether this file is a directory.

   modificationTime

   the file last modification time.

   blockReplication

   the block replication.

   blockSize

   the block size.

   action

   the file action. Must be either "add" or "delete".