Package org.apache.nifi.processor.util.file.transfer

Class FetchFileTransfer

java.lang.Object
org.apache.nifi.components.AbstractConfigurableComponent
org.apache.nifi.processor.AbstractSessionFactoryProcessor
org.apache.nifi.processor.AbstractProcessor
org.apache.nifi.processor.util.file.transfer.FetchFileTransfer
All Implemented Interfaces:
org.apache.nifi.components.ConfigurableComponent, org.apache.nifi.processor.Processor
public abstract class FetchFileTransfer extends org.apache.nifi.processor.AbstractProcessor
A base class for FetchSFTP, FetchFTP processors. Note that implementations of this class should never use the @SupportsBatching annotation! Doing so could result in data loss!

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    private static class 
    FetchFileTransfer.FileTransferIdleWrapper
    Wrapper around a FileTransfer object that is used to know when the FileTransfer was last used, so that we have the ability to close connections that are "idle," or unused for some period of time.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final org.apache.nifi.components.AllowableValue
    COMPLETION_DELETE
     
    static final org.apache.nifi.components.AllowableValue
    COMPLETION_MOVE
     
    static final org.apache.nifi.components.AllowableValue
    COMPLETION_NONE
     
    static final org.apache.nifi.components.PropertyDescriptor
    COMPLETION_STRATEGY
     
    static final String
    FAILURE_REASON_ATTRIBUTE
     
    static final org.apache.nifi.components.PropertyDescriptor
    FILE_NOT_FOUND_LOG_LEVEL
     
    private final Map<Tuple<String,Integer>,BlockingQueue<FetchFileTransfer.FileTransferIdleWrapper>>
    fileTransferMap
     
    static final org.apache.nifi.components.PropertyDescriptor
    HOSTNAME
     
    private final long
    IDLE_CONNECTION_MILLIS
     
    private long
    lastClearTime
     
    private org.apache.nifi.logging.LogLevel
    levelFileNotFound
     
    static final org.apache.nifi.components.PropertyDescriptor
    MOVE_CREATE_DIRECTORY
     
    static final org.apache.nifi.components.PropertyDescriptor
    MOVE_DESTINATION_DIR
     
    private static final List<org.apache.nifi.components.PropertyDescriptor>
    PROPERTY_DESCRIPTORS
     
    static final org.apache.nifi.processor.Relationship
    REL_COMMS_FAILURE
     
    static final org.apache.nifi.processor.Relationship
    REL_NOT_FOUND
     
    static final org.apache.nifi.processor.Relationship
    REL_PERMISSION_DENIED
     
    static final org.apache.nifi.processor.Relationship
    REL_SUCCESS
     
    private static final Set<org.apache.nifi.processor.Relationship>
    RELATIONSHIPS
     
    static final org.apache.nifi.components.PropertyDescriptor
    REMOTE_FILENAME
     
    static final org.apache.nifi.components.PropertyDescriptor
    UNDEFAULTED_PORT
     
    static final org.apache.nifi.components.PropertyDescriptor
    USERNAME
     

  • Constructor Summary

    Constructors
    Constructor
    Description
    FetchFileTransfer()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    cleanup()
     
    private void
    cleanupTransfer(FileTransfer transfer, boolean closeConnection, BlockingQueue<FetchFileTransfer.FileTransferIdleWrapper> transferQueue, String host, int port)
     
    private void
    closeConnections(boolean closeNonIdleConnections)
    Close connections that are idle or optionally close all connections.
    protected abstract FileTransfer
    createFileTransfer(org.apache.nifi.processor.ProcessContext context)
    Creates a new instance of a FileTransfer that can be used to pull files from a remote system.
    Set<org.apache.nifi.processor.Relationship>
    getRelationships()
     
    protected List<org.apache.nifi.components.PropertyDescriptor>
    getSupportedPropertyDescriptors()
     
    void
    onScheduled(org.apache.nifi.processor.ProcessContext context)
     
    void
    onTrigger(org.apache.nifi.processor.ProcessContext context, org.apache.nifi.processor.ProcessSession session)
     
    private void
    performCompletionStrategy(FileTransfer transfer, org.apache.nifi.processor.ProcessContext context, org.apache.nifi.flowfile.FlowFile flowFile, String filename, String host, int port)
     

    Methods inherited from class org.apache.nifi.processor.AbstractProcessor

    onTrigger

    Methods inherited from class org.apache.nifi.processor.AbstractSessionFactoryProcessor

    getControllerServiceLookup, getIdentifier, getLogger, getNodeTypeProvider, init, initialize, isConfigurationRestored, isScheduled, toString, updateConfiguredRestoredTrue, updateScheduledFalse, updateScheduledTrue

    Methods inherited from class org.apache.nifi.components.AbstractConfigurableComponent

    customValidate, equals, getPropertyDescriptor, getPropertyDescriptors, getSupportedDynamicPropertyDescriptor, hashCode, onPropertyModified, validate

    Methods inherited from class java.lang.Object

    clone, finalize, getClass, notify, notifyAll, wait, wait, wait

    Methods inherited from interface org.apache.nifi.components.ConfigurableComponent

    getPropertyDescriptor, getPropertyDescriptors, onPropertyModified, validate

    Methods inherited from interface org.apache.nifi.processor.Processor

    isStateful, migrateProperties, migrateRelationships

  • Field Details

    • COMPLETION_NONE

      public static final org.apache.nifi.components.AllowableValue COMPLETION_NONE

    • COMPLETION_MOVE

      public static final org.apache.nifi.components.AllowableValue COMPLETION_MOVE

    • COMPLETION_DELETE

      public static final org.apache.nifi.components.AllowableValue COMPLETION_DELETE

    • FAILURE_REASON_ATTRIBUTE

      public static final String FAILURE_REASON_ATTRIBUTE
      See Also:

    • HOSTNAME

      public static final org.apache.nifi.components.PropertyDescriptor HOSTNAME

    • UNDEFAULTED_PORT

      public static final org.apache.nifi.components.PropertyDescriptor UNDEFAULTED_PORT

    • USERNAME

      public static final org.apache.nifi.components.PropertyDescriptor USERNAME

    • REMOTE_FILENAME

      public static final org.apache.nifi.components.PropertyDescriptor REMOTE_FILENAME

    • COMPLETION_STRATEGY

      public static final org.apache.nifi.components.PropertyDescriptor COMPLETION_STRATEGY

    • MOVE_CREATE_DIRECTORY

      public static final org.apache.nifi.components.PropertyDescriptor MOVE_CREATE_DIRECTORY

    • MOVE_DESTINATION_DIR

      public static final org.apache.nifi.components.PropertyDescriptor MOVE_DESTINATION_DIR

    • FILE_NOT_FOUND_LOG_LEVEL

      public static final org.apache.nifi.components.PropertyDescriptor FILE_NOT_FOUND_LOG_LEVEL

    • REL_SUCCESS

      public static final org.apache.nifi.processor.Relationship REL_SUCCESS

    • REL_COMMS_FAILURE

      public static final org.apache.nifi.processor.Relationship REL_COMMS_FAILURE

    • REL_NOT_FOUND

      public static final org.apache.nifi.processor.Relationship REL_NOT_FOUND

    • REL_PERMISSION_DENIED

      public static final org.apache.nifi.processor.Relationship REL_PERMISSION_DENIED

    • PROPERTY_DESCRIPTORS

      private static final List<org.apache.nifi.components.PropertyDescriptor> PROPERTY_DESCRIPTORS

    • RELATIONSHIPS

      private static final Set<org.apache.nifi.processor.Relationship> RELATIONSHIPS

    • fileTransferMap

      private final Map<Tuple<String,Integer>,BlockingQueue<FetchFileTransfer.FileTransferIdleWrapper>> fileTransferMap

    • IDLE_CONNECTION_MILLIS

      private final long IDLE_CONNECTION_MILLIS

    • lastClearTime

      private volatile long lastClearTime

    • levelFileNotFound

      private org.apache.nifi.logging.LogLevel levelFileNotFound

  • Constructor Details

    • FetchFileTransfer

      public FetchFileTransfer()

  • Method Details

    • getRelationships

      public Set<org.apache.nifi.processor.Relationship> getRelationships()
      Specified by:
      getRelationships in interface org.apache.nifi.processor.Processor
      Overrides:
      getRelationships in class org.apache.nifi.processor.AbstractSessionFactoryProcessor

    • onScheduled

      @OnScheduled public void onScheduled(org.apache.nifi.processor.ProcessContext context)

    • closeConnections

      private void closeConnections(boolean closeNonIdleConnections)
      Close connections that are idle or optionally close all connections. Connections are considered "idle" if they have not been used in 10 seconds.
      Parameters:
      closeNonIdleConnections - if true will close all connection; if false will close only idle connections

    • cleanup

      @OnStopped public void cleanup()

    • getSupportedPropertyDescriptors

      protected List<org.apache.nifi.components.PropertyDescriptor> getSupportedPropertyDescriptors()
      Overrides:
      getSupportedPropertyDescriptors in class org.apache.nifi.components.AbstractConfigurableComponent

    • onTrigger

      public void onTrigger(org.apache.nifi.processor.ProcessContext context, org.apache.nifi.processor.ProcessSession session) throws org.apache.nifi.processor.exception.ProcessException
      Specified by:
      onTrigger in class org.apache.nifi.processor.AbstractProcessor
      Throws:
      org.apache.nifi.processor.exception.ProcessException

    • cleanupTransfer

      private void cleanupTransfer(FileTransfer transfer, boolean closeConnection, BlockingQueue<FetchFileTransfer.FileTransferIdleWrapper> transferQueue, String host, int port)

    • performCompletionStrategy

      private void performCompletionStrategy(FileTransfer transfer, org.apache.nifi.processor.ProcessContext context, org.apache.nifi.flowfile.FlowFile flowFile, String filename, String host, int port)

    • createFileTransfer

      protected abstract FileTransfer createFileTransfer(org.apache.nifi.processor.ProcessContext context)
      Creates a new instance of a FileTransfer that can be used to pull files from a remote system.
      Parameters:
      context - the ProcessContext to use in order to obtain configured properties
      Returns:
      a FileTransfer that can be used to pull files from a remote system