trait ScanFileListing extends AnyRef
Trait used to represent the selected partitions and dynamically selected partitions during file listing.
The ScanFileListing trait defines the core API for interacting with selected partitions,
establishing a contract for subclasses. It is situated at the root of this package and it is
designed to provide a widely accessible definition, that is accessible to other packages and
classes that need a way to represent the selected partitions and dynamically selected partitions.
- Alphabetic
- By Inheritance
- ScanFileListing
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Abstract Value Members
- abstract def bucketsContainSingleFile: Boolean
Determines if each bucket in the current file listing representation contains at most one file.
Determines if each bucket in the current file listing representation contains at most one file. This function returns true if it does, or false otherwise.
- abstract def calculateTotalPartitionBytes: Long
Returns the total partition size in bytes for the current ScanFileListing representation.
- abstract def filePartitionIterator: Iterator[ListingPartition]
Returns an iterator of over the partitions and their files for the file listing representation.
Returns an iterator of over the partitions and their files for the file listing representation. This allows us to iterate over the partitions without the additional overhead of materializing the whole collection.
- abstract def filterAndPruneFiles(boundPredicate: BasePredicate, dynamicFileFilters: Seq[Expression]): ScanFileListing
Filters and prunes files from the current scan file listing representation based on the given predicate and dynamic file filters.
Filters and prunes files from the current scan file listing representation based on the given predicate and dynamic file filters. Initially, it filters partitions based on a static predicate. For partitions that pass this filter, it further prunes files using dynamic file filters, if any are provided. This method assumes that dynamic file filters are applicable only to files within partitions that have already passed the static predicate filter.
- abstract def partitionCount: Int
Returns the number of partitions for the current partition representation.
- abstract def toPartitionArray: Array[PartitionedFile]
Returns an Array[PartitionedFile from the current ScanFileListing representation.
- abstract def totalFileSize: Long
Calculates the total size in bytes of all files across the current file listing representation.
- abstract def totalNumberOfFiles: Long
Returns the total number of files across the current file listing representation.
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)