Interface FileSplitAssigner
-
- All Known Implementing Classes:
LocalityAwareSplitAssigner,SimpleSplitAssigner
@PublicEvolving public interface FileSplitAssignerTheFileSplitAssigneris responsible for deciding what split should be processed next by which node. It determines split processing order and locality.
-
-
Nested Class Summary
Nested Classes Modifier and Type Interface Description static interfaceFileSplitAssigner.ProviderFactory for theFileSplitAssigner, to allow theFileSplitAssignerto be eagerly initialized and to not be serializable.
-
Method Summary
All Methods Instance Methods Abstract Methods Modifier and Type Method Description voidaddSplits(Collection<FileSourceSplit> splits)Adds a set of splits to this assigner.Optional<FileSourceSplit>getNext(String hostname)Gets the next split.Collection<FileSourceSplit>remainingSplits()Gets the remaining splits that this assigner has pending.
-
-
-
Method Detail
-
getNext
Optional<FileSourceSplit> getNext(@Nullable String hostname)
Gets the next split.When this method returns an empty
Optional, then the set of splits is assumed to be done and the source will finish once the readers finished their current splits.
-
addSplits
void addSplits(Collection<FileSourceSplit> splits)
Adds a set of splits to this assigner. This happens for example when some split processing failed and the splits need to be re-added, or when new splits got discovered.
-
remainingSplits
Collection<FileSourceSplit> remainingSplits()
Gets the remaining splits that this assigner has pending.
-
-