Package com.google.cloud.hadoop.fs.gcs

Class GoogleHadoopFS

java.lang.Object
org.apache.hadoop.fs.AbstractFileSystem
com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS
All Implemented Interfaces:
org.apache.hadoop.fs.PathCapabilities
public class GoogleHadoopFS extends org.apache.hadoop.fs.AbstractFileSystem
GoogleHadoopFS provides a YARN compatible Abstract File System on top of Google Cloud Storage (GCS).

It is implemented as a thin abstraction layer on top of GoogleHadoopFileSystem, but will soon be refactored to share a common base.

  • Field Summary

    Fields inherited from class org.apache.hadoop.fs.AbstractFileSystem

    statistics

  • Constructor Summary

    Constructors
    Constructor
    Description
    GoogleHadoopFS(GoogleHadoopFileSystem ghfs, URI uri, org.apache.hadoop.conf.Configuration conf)
     
    GoogleHadoopFS(URI uri, org.apache.hadoop.conf.Configuration conf)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    checkPath(org.apache.hadoop.fs.Path path)
    Only accept valid AbstractFileSystem and GoogleHadoopFileSystem Paths.
    org.apache.hadoop.fs.FSDataOutputStream
    createInternal(org.apache.hadoop.fs.Path file, EnumSet<org.apache.hadoop.fs.CreateFlag> flag, org.apache.hadoop.fs.permission.FsPermission absolutePermission, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress, org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt, boolean createParent)
     
    boolean
    delete(org.apache.hadoop.fs.Path f, boolean recursive)
     
    org.apache.hadoop.fs.BlockLocation[]
    getFileBlockLocations(org.apache.hadoop.fs.Path f, long start, long len)
     
    org.apache.hadoop.fs.FileChecksum
    getFileChecksum(org.apache.hadoop.fs.Path f)
     
    org.apache.hadoop.fs.FileStatus
    getFileStatus(org.apache.hadoop.fs.Path f)
     
    org.apache.hadoop.fs.FsStatus
    getFsStatus()
     
    org.apache.hadoop.fs.FsServerDefaults
    getServerDefaults()
     
    URI
    getUri()
    This is overridden to use GoogleHadoopFileSystem's URI, because AbstractFileSystem appends the default port to the authority.
    int
    getUriDefaultPort()
     
    boolean
    isValidName(String src)
    Follow HDFS conventions except allow for ':' in paths.
    org.apache.hadoop.fs.FileStatus[]
    listStatus(org.apache.hadoop.fs.Path f)
     
    void
    mkdir(org.apache.hadoop.fs.Path dir, org.apache.hadoop.fs.permission.FsPermission permission, boolean createParent)
     
    org.apache.hadoop.fs.FSDataInputStream
    open(org.apache.hadoop.fs.Path f, int bufferSize)
     
    void
    renameInternal(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)
     
    void
    setOwner(org.apache.hadoop.fs.Path f, String username, String groupname)
     
    void
    setPermission(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission)
     
    boolean
    setReplication(org.apache.hadoop.fs.Path f, short replication)
     
    void
    setTimes(org.apache.hadoop.fs.Path f, long mtime, long atime)
     
    void
    setVerifyChecksum(boolean verifyChecksum)
     

    Methods inherited from class org.apache.hadoop.fs.AbstractFileSystem

    access, checkScheme, clearStatistics, create, createFileSystem, createMultipartUploader, createSnapshot, createSymlink, deleteSnapshot, equals, get, getAclStatus, getAllStatistics, getAllStoragePolicies, getCanonicalServiceName, getDelegationTokens, getEnclosingRoot, getFileLinkStatus, getFsStatus, getHomeDirectory, getInitialWorkingDirectory, getLinkTarget, getServerDefaults, getStatistics, getStatistics, getStoragePolicy, getUriPath, getXAttr, getXAttrs, getXAttrs, hashCode, hasPathCapability, listCorruptFileBlocks, listLocatedStatus, listStatusIterator, listXAttrs, makeQualified, methodNotSupported, modifyAclEntries, msync, open, openFileWithOptions, printStatistics, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameInternal, renameSnapshot, resolvePath, satisfyStoragePolicy, setAcl, setStoragePolicy, setXAttr, setXAttr, supportsSymlinks, truncate, unsetStoragePolicy

    Methods inherited from class java.lang.Object

    clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

  • Method Details

    • createInternal

      public org.apache.hadoop.fs.FSDataOutputStream createInternal(org.apache.hadoop.fs.Path file, EnumSet<org.apache.hadoop.fs.CreateFlag> flag, org.apache.hadoop.fs.permission.FsPermission absolutePermission, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress, org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt, boolean createParent) throws IOException
      Specified by:
      createInternal in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      IOException

    • getUriDefaultPort

      public int getUriDefaultPort()
      Specified by:
      getUriDefaultPort in class org.apache.hadoop.fs.AbstractFileSystem

    • getUri

      public URI getUri()
      This is overridden to use GoogleHadoopFileSystem's URI, because AbstractFileSystem appends the default port to the authority.
      Overrides:
      getUri in class org.apache.hadoop.fs.AbstractFileSystem

    • isValidName

      public boolean isValidName(String src)
      Follow HDFS conventions except allow for ':' in paths.
      Overrides:
      isValidName in class org.apache.hadoop.fs.AbstractFileSystem

    • checkPath

      public void checkPath(org.apache.hadoop.fs.Path path)
      Only accept valid AbstractFileSystem and GoogleHadoopFileSystem Paths.
      Overrides:
      checkPath in class org.apache.hadoop.fs.AbstractFileSystem

    • getServerDefaults

      public org.apache.hadoop.fs.FsServerDefaults getServerDefaults() throws IOException
      Specified by:
      getServerDefaults in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      IOException

    • mkdir

      public void mkdir(org.apache.hadoop.fs.Path dir, org.apache.hadoop.fs.permission.FsPermission permission, boolean createParent) throws IOException
      Specified by:
      mkdir in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      IOException

    • delete

      public boolean delete(org.apache.hadoop.fs.Path f, boolean recursive) throws IOException
      Specified by:
      delete in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      IOException

    • open

      public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path f, int bufferSize) throws IOException
      Specified by:
      open in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      IOException

    • setReplication

      public boolean setReplication(org.apache.hadoop.fs.Path f, short replication) throws IOException
      Specified by:
      setReplication in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      IOException

    • renameInternal

      public void renameInternal(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst) throws IOException
      Specified by:
      renameInternal in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      IOException

    • setPermission

      public void setPermission(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission) throws IOException
      Specified by:
      setPermission in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      IOException

    • setOwner

      public void setOwner(org.apache.hadoop.fs.Path f, String username, String groupname) throws IOException
      Specified by:
      setOwner in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      IOException

    • setTimes

      public void setTimes(org.apache.hadoop.fs.Path f, long mtime, long atime) throws IOException
      Specified by:
      setTimes in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      IOException

    • getFileChecksum

      public org.apache.hadoop.fs.FileChecksum getFileChecksum(org.apache.hadoop.fs.Path f) throws IOException
      Specified by:
      getFileChecksum in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      IOException

    • getFileStatus

      public org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.fs.Path f) throws IOException
      Specified by:
      getFileStatus in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      IOException

    • getFileBlockLocations

      public org.apache.hadoop.fs.BlockLocation[] getFileBlockLocations(org.apache.hadoop.fs.Path f, long start, long len) throws IOException
      Specified by:
      getFileBlockLocations in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      IOException

    • getFsStatus

      public org.apache.hadoop.fs.FsStatus getFsStatus() throws IOException
      Specified by:
      getFsStatus in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      IOException

    • listStatus

      public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path f) throws IOException
      Specified by:
      listStatus in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      IOException

    • setVerifyChecksum

      public void setVerifyChecksum(boolean verifyChecksum)
      Specified by:
      setVerifyChecksum in class org.apache.hadoop.fs.AbstractFileSystem