Package org.apache.nifi.processors.standard

Class SplitText

java.lang.Object
org.apache.nifi.components.AbstractConfigurableComponent
org.apache.nifi.processor.AbstractSessionFactoryProcessor
org.apache.nifi.processor.AbstractProcessor
org.apache.nifi.processors.standard.SplitText
All Implemented Interfaces:
org.apache.nifi.components.ConfigurableComponent, org.apache.nifi.processor.Processor
@SideEffectFree @SupportsBatching @Tags({"split","text"}) @InputRequirement(INPUT_REQUIRED) @CapabilityDescription("Splits a text file into multiple smaller text files on line boundaries limited by maximum number of lines or total size of fragment. Each output split file will contain no more than the configured number of lines or bytes. If both Line Split Count and Maximum Fragment Size are specified, the split occurs at whichever limit is reached first. If the first line of a fragment exceeds the Maximum Fragment Size, that line will be output in a single split file which exceeds the configured maximum size limit. This component also allows one to specify that each split should include a header lines. Header lines can be computed by either specifying the amount of lines that should constitute a header or by using header marker to match against the read lines. If such match happens then the corresponding line will be treated as header. Keep in mind that upon the first failure of header marker match, no more matches will be performed and the rest of the data will be parsed as regular lines for a given split. If after computation of the header there are no more data, the resulting split will consists of only header lines.") @WritesAttribute(attribute="text.line.count",description="The number of lines of text from the original FlowFile that were copied to this FlowFile") @WritesAttribute(attribute="fragment.size",description="The number of bytes from the original FlowFile that were copied to this FlowFile, including header, if applicable, which is duplicated in each split FlowFile") @WritesAttribute(attribute="fragment.identifier",description="All split FlowFiles produced from the same parent FlowFile will have the same randomly generated UUID added for this attribute") @WritesAttribute(attribute="fragment.index",description="A one-up number that indicates the ordering of the split FlowFiles that were created from a single parent FlowFile") @WritesAttribute(attribute="fragment.count",description="The number of split FlowFiles generated from the parent FlowFile") @WritesAttribute(attribute="segment.original.filename ",description="The filename of the parent FlowFile") @SeeAlso(MergeContent.class) @SystemResourceConsideration(resource=MEMORY, description="The FlowFile with its attributes is stored in memory, not the content of the FlowFile. If many splits are generated due to the size of the content, or how the content is configured to be split, a two-phase approach may be necessary to avoid excessive use of memory.") public class SplitText extends org.apache.nifi.processor.AbstractProcessor

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    private class 
    SplitText.SplitInfo
    Container for hosting meta-information pertaining to the split so it can be used later to create FlowFile representing the split.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final String
    FRAGMENT_COUNT
     
    static final String
    FRAGMENT_ID
     
    static final String
    FRAGMENT_INDEX
     
    static final org.apache.nifi.components.PropertyDescriptor
    FRAGMENT_MAX_SIZE
     
    static final String
    FRAGMENT_SIZE
     
    static final org.apache.nifi.components.PropertyDescriptor
    HEADER_LINE_COUNT
     
    static final org.apache.nifi.components.PropertyDescriptor
    HEADER_MARKER
     
    private int
    headerLineCount
     
    private String
    headerMarker
     
    static final org.apache.nifi.components.PropertyDescriptor
    LINE_SPLIT_COUNT
     
    private int
    lineCount
     
    private long
    maxSplitSize
     
    private static final List<org.apache.nifi.components.PropertyDescriptor>
    PROPERTY_DESCRIPTORS
     
    static final org.apache.nifi.processor.Relationship
    REL_FAILURE
     
    static final org.apache.nifi.processor.Relationship
    REL_ORIGINAL
     
    static final org.apache.nifi.processor.Relationship
    REL_SPLITS
     
    private static final Set<org.apache.nifi.processor.Relationship>
    RELATIONSHIPS
     
    static final org.apache.nifi.components.PropertyDescriptor
    REMOVE_TRAILING_NEWLINES
     
    private boolean
    removeTrailingNewLines
     
    static final String
    SEGMENT_ORIGINAL_FILENAME
     
    static final String
    SPLIT_LINE_COUNT
     

  • Constructor Summary

    Constructors
    Constructor
    Description
    SplitText()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    private SplitText.SplitInfo
    computeHeader(TextLineDemarcator demarcator, long startOffset, long splitMaxLineCount, byte[] startsWithFilter, SplitText.SplitInfo previousSplitInfo)
    Will generate SplitText.SplitInfo for the next fragment that represents the header of the future split.
    private org.apache.nifi.flowfile.FlowFile
    concatenateContents(org.apache.nifi.flowfile.FlowFile sourceFlowFile, org.apache.nifi.processor.ProcessSession session, org.apache.nifi.flowfile.FlowFile... flowFiles)
    Will concatenate the contents of the provided array of FlowFiles into a single FlowFile.
    protected Collection<org.apache.nifi.components.ValidationResult>
    customValidate(org.apache.nifi.components.ValidationContext validationContext)
     
    private List<org.apache.nifi.flowfile.FlowFile>
    generateSplitFlowFiles(String fragmentId, org.apache.nifi.flowfile.FlowFile sourceFlowFile, SplitText.SplitInfo splitInfo, List<SplitText.SplitInfo> computedSplitsInfo, org.apache.nifi.processor.ProcessSession processSession)
    Generates the list of FlowFiles representing splits.
    Set<org.apache.nifi.processor.Relationship>
    getRelationships()
     
    protected List<org.apache.nifi.components.PropertyDescriptor>
    getSupportedPropertyDescriptors()
     
    private SplitText.SplitInfo
    nextSplit(TextLineDemarcator demarcator, long startOffset, long splitMaxLineCount, SplitText.SplitInfo remainderSplitInfo, long startingLength)
    Will generate SplitText.SplitInfo for the next split.
    void
    onSchedule(org.apache.nifi.processor.ProcessContext context)
     
    void
    onTrigger(org.apache.nifi.processor.ProcessContext context, org.apache.nifi.processor.ProcessSession processSession)
    Will split the incoming stream releasing all splits as FlowFile at once.
    private org.apache.nifi.flowfile.FlowFile
    updateAttributes(org.apache.nifi.processor.ProcessSession processSession, org.apache.nifi.flowfile.FlowFile splitFlowFile, long splitLineCount, long splitFlowFileSize, String splitId, int splitIndex, String origFileName)
     

    Methods inherited from class org.apache.nifi.processor.AbstractProcessor

    onTrigger

    Methods inherited from class org.apache.nifi.processor.AbstractSessionFactoryProcessor

    getControllerServiceLookup, getIdentifier, getLogger, getNodeTypeProvider, init, initialize, isConfigurationRestored, isScheduled, toString, updateConfiguredRestoredTrue, updateScheduledFalse, updateScheduledTrue

    Methods inherited from class org.apache.nifi.components.AbstractConfigurableComponent

    equals, getPropertyDescriptor, getPropertyDescriptors, getSupportedDynamicPropertyDescriptor, hashCode, onPropertyModified, validate

    Methods inherited from class java.lang.Object

    clone, finalize, getClass, notify, notifyAll, wait, wait, wait

    Methods inherited from interface org.apache.nifi.components.ConfigurableComponent

    getPropertyDescriptor, getPropertyDescriptors, onPropertyModified, validate

    Methods inherited from interface org.apache.nifi.processor.Processor

    isStateful, migrateProperties, migrateRelationships

  • Field Details

    • SPLIT_LINE_COUNT

      public static final String SPLIT_LINE_COUNT
      See Also:

    • FRAGMENT_SIZE

      public static final String FRAGMENT_SIZE

    • FRAGMENT_ID

      public static final String FRAGMENT_ID

    • FRAGMENT_INDEX

      public static final String FRAGMENT_INDEX

    • FRAGMENT_COUNT

      public static final String FRAGMENT_COUNT

    • SEGMENT_ORIGINAL_FILENAME

      public static final String SEGMENT_ORIGINAL_FILENAME

    • LINE_SPLIT_COUNT

      public static final org.apache.nifi.components.PropertyDescriptor LINE_SPLIT_COUNT

    • FRAGMENT_MAX_SIZE

      public static final org.apache.nifi.components.PropertyDescriptor FRAGMENT_MAX_SIZE

    • HEADER_LINE_COUNT

      public static final org.apache.nifi.components.PropertyDescriptor HEADER_LINE_COUNT

    • HEADER_MARKER

      public static final org.apache.nifi.components.PropertyDescriptor HEADER_MARKER

    • REMOVE_TRAILING_NEWLINES

      public static final org.apache.nifi.components.PropertyDescriptor REMOVE_TRAILING_NEWLINES

    • PROPERTY_DESCRIPTORS

      private static final List<org.apache.nifi.components.PropertyDescriptor> PROPERTY_DESCRIPTORS

    • REL_ORIGINAL

      public static final org.apache.nifi.processor.Relationship REL_ORIGINAL

    • REL_SPLITS

      public static final org.apache.nifi.processor.Relationship REL_SPLITS

    • REL_FAILURE

      public static final org.apache.nifi.processor.Relationship REL_FAILURE

    • RELATIONSHIPS

      private static final Set<org.apache.nifi.processor.Relationship> RELATIONSHIPS

    • removeTrailingNewLines

      private volatile boolean removeTrailingNewLines

    • maxSplitSize

      private volatile long maxSplitSize

    • lineCount

      private volatile int lineCount

    • headerLineCount

      private volatile int headerLineCount

    • headerMarker

      private volatile String headerMarker

  • Constructor Details

    • SplitText

      public SplitText()

  • Method Details

    • getRelationships

      public Set<org.apache.nifi.processor.Relationship> getRelationships()
      Specified by:
      getRelationships in interface org.apache.nifi.processor.Processor
      Overrides:
      getRelationships in class org.apache.nifi.processor.AbstractSessionFactoryProcessor

    • onSchedule

      @OnScheduled public void onSchedule(org.apache.nifi.processor.ProcessContext context)

    • onTrigger

      public void onTrigger(org.apache.nifi.processor.ProcessContext context, org.apache.nifi.processor.ProcessSession processSession) throws org.apache.nifi.processor.exception.ProcessException
      Will split the incoming stream releasing all splits as FlowFile at once.
      Specified by:
      onTrigger in class org.apache.nifi.processor.AbstractProcessor
      Throws:
      org.apache.nifi.processor.exception.ProcessException

    • customValidate

      protected Collection<org.apache.nifi.components.ValidationResult> customValidate(org.apache.nifi.components.ValidationContext validationContext)
      Overrides:
      customValidate in class org.apache.nifi.components.AbstractConfigurableComponent

    • getSupportedPropertyDescriptors

      protected List<org.apache.nifi.components.PropertyDescriptor> getSupportedPropertyDescriptors()
      Overrides:
      getSupportedPropertyDescriptors in class org.apache.nifi.components.AbstractConfigurableComponent

    • generateSplitFlowFiles

      private List<org.apache.nifi.flowfile.FlowFile> generateSplitFlowFiles(String fragmentId, org.apache.nifi.flowfile.FlowFile sourceFlowFile, SplitText.SplitInfo splitInfo, List<SplitText.SplitInfo> computedSplitsInfo, org.apache.nifi.processor.ProcessSession processSession)
      Generates the list of FlowFiles representing splits. If SplitText.SplitInfo provided as an argument to this operation is not null it signifies the header information and its contents will be included in each and every computed split.

    • concatenateContents

      private org.apache.nifi.flowfile.FlowFile concatenateContents(org.apache.nifi.flowfile.FlowFile sourceFlowFile, org.apache.nifi.processor.ProcessSession session, org.apache.nifi.flowfile.FlowFile... flowFiles)
      Will concatenate the contents of the provided array of FlowFiles into a single FlowFile. While this operation is as general as it is described in the previous sentence, in the context of this processor there can only be two FlowFiles with the first FlowFile representing the header content of the split and the second FlowFile represents the split itself.

    • updateAttributes

      private org.apache.nifi.flowfile.FlowFile updateAttributes(org.apache.nifi.processor.ProcessSession processSession, org.apache.nifi.flowfile.FlowFile splitFlowFile, long splitLineCount, long splitFlowFileSize, String splitId, int splitIndex, String origFileName)

    • computeHeader

      private SplitText.SplitInfo computeHeader(TextLineDemarcator demarcator, long startOffset, long splitMaxLineCount, byte[] startsWithFilter, SplitText.SplitInfo previousSplitInfo) throws IOException
      Will generate SplitText.SplitInfo for the next fragment that represents the header of the future split.

      If split size is controlled by the amount of lines in the split then the resulting SplitText.SplitInfo line count will always be invalid input: '<'= 'splitMaxLineCount'. It can only be less IF it reaches the EOF. If split size is controlled by the maxSplitSize, then the resulting SplitText.SplitInfo line count will vary but the length of the split will never be > maxSplitSize and IllegalStateException will be thrown. This method also allows one to provide 'startsWithFilter' to allow headers to be determined via such filter (see HEADER_MARKER.

      Throws:
      IOException

    • nextSplit

      private SplitText.SplitInfo nextSplit(TextLineDemarcator demarcator, long startOffset, long splitMaxLineCount, SplitText.SplitInfo remainderSplitInfo, long startingLength) throws IOException
      Will generate SplitText.SplitInfo for the next split.

      If split size is controlled by the amount of lines in the split then the resulting SplitText.SplitInfo line count will always be invalid input: '<'= 'splitMaxLineCount'. If split size is controlled by the maxSplitSize, then the resulting SplitText.SplitInfo line count will vary but the length of the split will never be > maxSplitSize.

      Throws:
      IOException