org.apache.hadoop.fs

Class FileContext

java.lang.Object

org.apache.hadoop.fs.FileContext

@InterfaceAudience.Public
 @InterfaceStability.Stable
public class FileContext
extends Object

The FileContext class provides an interface for users of the Hadoop file system. It exposes a number of file system operations, e.g. create, open, list.

Path Names

The Hadoop file system supports a URI namespace and URI names. This enables multiple types of file systems to be referenced using fully-qualified URIs. Two common Hadoop file system implementations are

• the local file system: file:///path
• the HDFS file system: hdfs://nnAddress:nnPort/path

The Hadoop file system also supports additional naming schemes besides URIs. Hadoop has the concept of a default file system, which implies a default URI scheme and authority. This enables slash-relative names relative to the default FS, which are more convenient for users and application writers. The default FS is typically set by the user's environment, though it can also be manually specified.

Hadoop also supports working-directory-relative names, which are paths relative to the current working directory (similar to Unix). The working directory can be in a different file system than the default FS.

Thus, Hadoop path names can be specified as one of the following:

• a fully-qualified URI: scheme://authority/path (e.g. hdfs://nnAddress:nnPort/foo/bar)
• a slash-relative name: path relative to the default file system (e.g. /foo/bar)
• a working-directory-relative name: path relative to the working dir (e.g. foo/bar)

Relative paths with scheme (scheme:foo/bar) are illegal.

Role of FileContext and Configuration Defaults

The FileContext is the analogue of per-process file-related state in Unix. It contains two properties:

• the default file system (for resolving slash-relative names)
• the umask (for file permissions)

In general, these properties are obtained from the default configuration file in the user's environment (see Configuration). Further file system properties are specified on the server-side. File system operations default to using these server-side defaults unless otherwise specified.

The file system related server-side defaults are:

• the home directory (default is "/user/userName")
• the initial wd (only for local fs)
• replication factor
• block size
• buffer size
• encryptDataTransfer
• checksum option. (checksumType and bytesPerChecksum)

Example Usage

Example 1: use the default config read from the $HADOOP_CONFIG/core.xml. Unspecified values come from core-defaults.xml in the release jar.

• myFContext = FileContext.getFileContext(); // uses the default config // which has your default FS
• myFContext.create(path, ...);
• myFContext.setWorkingDir(path);
• myFContext.open (path, ...);
• ...

Example 2: Get a FileContext with a specific URI as the default FS

• myFContext = FileContext.getFileContext(URI);
• myFContext.create(path, ...);
• ...

Example 3: FileContext with local file system as the default

• myFContext = FileContext.getLocalFSFileContext();
• myFContext.create(path, ...);
• ...

Example 4: Use a specific config, ignoring $HADOOP_CONFIG Generally you should not need use a config unless you are doing

• configX = someConfigSomeOnePassedToYou;
• myFContext = getFileContext(configX); // configX is not changed, // is passed down
• myFContext.create(path, ...);
• ...

Field Summary

Fields

Modifier and Type	Field and Description
static FsPermission	DEFAULT_PERM Default permission for directory and symlink In previous versions, this default permission was also used to create files, so files created end up with ugo+x permission.
static FsPermission	DIR_DEFAULT_PERM Default permission for directory
static FsPermission	FILE_DEFAULT_PERM Default permission for file
static org.slf4j.Logger	LOG
static int	SHUTDOWN_HOOK_PRIORITY Priority of the FileContext shutdown hook.

Method Summary

Modifier and Type	Method and Description
static void	clearStatistics() Clears all the statistics stored in AbstractFileSystem, for all the file systems.
FSDataOutputStreamBuilder<FSDataOutputStream,?>	create(Path f) Create a FSDataOutputStreamBuilder for creating or overwriting a file on indicated path.
FSDataOutputStream	create(Path f, EnumSet<CreateFlag> createFlag, org.apache.hadoop.fs.Options.CreateOpts... opts) Create or overwrite file on indicated path and returns an output stream for writing into the file.
Path	createSnapshot(Path path) Create a snapshot with a default name.
Path	createSnapshot(Path path, String snapshotName) Create a snapshot.
void	createSymlink(Path target, Path link, boolean createParent) Creates a symbolic link to an existing file.
boolean	delete(Path f, boolean recursive) Delete a file.
boolean	deleteOnExit(Path f) Mark a path to be deleted on JVM shutdown.
void	deleteSnapshot(Path path, String snapshotName) Delete a snapshot of a directory.
AclStatus	getAclStatus(Path path) Gets the ACLs of files and directories.
static Map<URI,org.apache.hadoop.fs.FileSystem.Statistics>	getAllStatistics()
Collection<? extends BlockStoragePolicySpi>	getAllStoragePolicies() Retrieve all the storage policies supported by this file system.
FileChecksum	getFileChecksum(Path f) Get the checksum of a file.
static FileContext	getFileContext() Create a FileContext using the default config read from the $HADOOP_CONFIG/core.xml, Unspecified key-values for config are defaulted from core-defaults.xml in the release jar.
protected static FileContext	getFileContext(AbstractFileSystem defaultFS) Create a FileContext for specified file system using the default config.
static FileContext	getFileContext(AbstractFileSystem defFS, Configuration aConf) Create a FileContext with specified FS as default using the specified config.
static FileContext	getFileContext(Configuration aConf) Create a FileContext using the passed config.
static FileContext	getFileContext(URI defaultFsUri) Create a FileContext for specified URI using the default config.
static FileContext	getFileContext(URI defaultFsUri, Configuration aConf) Create a FileContext for specified default URI using the specified config.
FileStatus	getFileLinkStatus(Path f) Return a file status object that represents the path.
FileStatus	getFileStatus(Path f) Return a file status object that represents the path.
protected AbstractFileSystem	getFSofPath(Path absOrFqPath) Get the file system of supplied path.
FsStatus	getFsStatus(Path f) Returns a status object describing the use and capacity of the file system denoted by the Parh argument p.
Path	getHomeDirectory() Return the current user's home directory in this file system.
Path	getLinkTarget(Path f) Returns the target of the given symbolic link as it was specified when the link was created.
static FileContext	getLocalFSFileContext()
static FileContext	getLocalFSFileContext(Configuration aConf)
static org.apache.hadoop.fs.FileSystem.Statistics	getStatistics(URI uri) Get the statistics for a particular file system
BlockStoragePolicySpi	getStoragePolicy(Path path) Query the effective storage policy ID for the given file or directory.
UserGroupInformation	getUgi() Gets the ugi in the file-context
FsPermission	getUMask()
Path	getWorkingDirectory() Gets the working directory for wd-relative names (such a "foo/bar").
byte[]	getXAttr(Path path, String name) Get an xattr for a file or directory.
Map<String,byte[]>	getXAttrs(Path path) Get all of the xattrs for a file or directory.
Map<String,byte[]>	getXAttrs(Path path, List<String> names) Get all of the xattrs for a file or directory.
org.apache.hadoop.fs.RemoteIterator<Path>	listCorruptFileBlocks(Path path)
org.apache.hadoop.fs.RemoteIterator<LocatedFileStatus>	listLocatedStatus(Path f) List the statuses of the files/directories in the given path if the path is a directory.
org.apache.hadoop.fs.RemoteIterator<FileStatus>	listStatus(Path f) List the statuses of the files/directories in the given path if the path is a directory.
List<String>	listXAttrs(Path path) Get all of the xattr names for a file or directory.
Path	makeQualified(Path path) Make the path fully qualified if it is isn't.
void	mkdir(Path dir, FsPermission permission, boolean createParent) Make(create) a directory and all the non-existent parents.
void	modifyAclEntries(Path path, List<AclEntry> aclSpec) Modifies ACL entries of files and directories.
FSDataInputStream	open(Path f) Opens an FSDataInputStream at the indicated Path using default buffersize.
FSDataInputStream	open(Path f, int bufferSize) Opens an FSDataInputStream at the indicated Path.
FutureDataInputStreamBuilder	openFile(Path path) Open a file for reading through a builder API.
static void	printStatistics() Prints the statistics to standard output.
void	removeAcl(Path path) Removes all but the base ACL entries of files and directories.
void	removeAclEntries(Path path, List<AclEntry> aclSpec) Removes ACL entries from files and directories.
void	removeDefaultAcl(Path path) Removes all default ACL entries from files and directories.
void	removeXAttr(Path path, String name) Remove an xattr of a file or directory.
void	rename(Path src, Path dst, org.apache.hadoop.fs.Options.Rename... options) Renames Path src to Path dst Fails if src is a file and dst is a directory.
void	renameSnapshot(Path path, String snapshotOldName, String snapshotNewName) Rename a snapshot.
protected Path	resolve(Path f) Resolves all symbolic links in the specified path.
protected Path	resolveIntermediate(Path f) Resolves all symbolic links in the specified path leading up to, but not including the final path component.
Path	resolvePath(Path f) Resolve the path following any symlinks or mount points
void	satisfyStoragePolicy(Path path) Set the source path to satisfy storage policy.
void	setAcl(Path path, List<AclEntry> aclSpec) Fully replaces ACL of files and directories, discarding all existing entries.
void	setOwner(Path f, String username, String groupname) Set owner of a path (i.e.
void	setPermission(Path f, FsPermission permission) Set permission of a path.
boolean	setReplication(Path f, short replication) Set replication for an existing file.
void	setStoragePolicy(Path path, String policyName) Set the storage policy for a given file or directory.
void	setTimes(Path f, long mtime, long atime) Set access time of a file.
void	setUMask(FsPermission newUmask) Set umask to the supplied parameter.
void	setVerifyChecksum(boolean verifyChecksum, Path f) Set the verify checksum flag for the file system denoted by the path.
void	setWorkingDirectory(Path newWDir) Set the working directory for wd-relative names (such a "foo/bar").
void	setXAttr(Path path, String name, byte[] value) Set an xattr of a file or directory.
void	setXAttr(Path path, String name, byte[] value, EnumSet<XAttrSetFlag> flag) Set an xattr of a file or directory.
boolean	truncate(Path f, long newLength) Truncate the file in the indicated path to the indicated size.
void	unsetStoragePolicy(Path src) Unset the storage policy set for a given file or directory.
org.apache.hadoop.fs.FileContext.Util	util()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

DEFAULT_PERM

public static final FsPermission DEFAULT_PERM

Default permission for directory and symlink In previous versions, this default permission was also used to create files, so files created end up with ugo+x permission. See HADOOP-9155 for detail. Two new constants are added to solve this, please use DIR_DEFAULT_PERM for directory, and use FILE_DEFAULT_PERM for file. This constant is kept for compatibility.

DIR_DEFAULT_PERM

public static final FsPermission DIR_DEFAULT_PERM

Default permission for directory

FILE_DEFAULT_PERM

public static final FsPermission FILE_DEFAULT_PERM

Default permission for file

SHUTDOWN_HOOK_PRIORITY

public static final int SHUTDOWN_HOOK_PRIORITY

Priority of the FileContext shutdown hook.

See Also:

Constant Field Values

Method Detail

getFSofPath

protected AbstractFileSystem getFSofPath(Path absOrFqPath)
                                  throws UnsupportedFileSystemException,
                                         IOException

Get the file system of supplied path.

Parameters:

absOrFqPath - - absolute or fully qualified path

Returns:

the file system of the path

Throws:

UnsupportedFileSystemException - If the file system for absOrFqPath is not supported.

IOException - If the file system for absOrFqPath could not be instantiated.

getFileContext

public static FileContext getFileContext(AbstractFileSystem defFS,
                                         Configuration aConf)

Create a FileContext with specified FS as default using the specified config.

Parameters:

defFS -

aConf -

Returns:

new FileContext with specified FS as default.

getFileContext

protected static FileContext getFileContext(AbstractFileSystem defaultFS)

Create a FileContext for specified file system using the default config.

Parameters:

defaultFS -

Returns:

a FileContext with the specified AbstractFileSystem as the default FS.

getFileContext

public static FileContext getFileContext()
                                  throws UnsupportedFileSystemException

Create a FileContext using the default config read from the $HADOOP_CONFIG/core.xml, Unspecified key-values for config are defaulted from core-defaults.xml in the release jar.

Throws:

UnsupportedFileSystemException - If the file system from the default configuration is not supported

getLocalFSFileContext

public static FileContext getLocalFSFileContext()
                                         throws UnsupportedFileSystemException

Returns:

a FileContext for the local file system using the default config.

Throws:

UnsupportedFileSystemException - If the file system for FsConstants.LOCAL_FS_URI is not supported.

getFileContext

public static FileContext getFileContext(URI defaultFsUri)
                                  throws UnsupportedFileSystemException

Create a FileContext for specified URI using the default config.

Parameters:

defaultFsUri -

Returns:

a FileContext with the specified URI as the default FS.

Throws:

UnsupportedFileSystemException - If the file system for defaultFsUri is not supported

getFileContext

public static FileContext getFileContext(URI defaultFsUri,
                                         Configuration aConf)
                                  throws UnsupportedFileSystemException

Create a FileContext for specified default URI using the specified config.

Parameters:

defaultFsUri -

aConf -

Returns:

new FileContext for specified uri

Throws:

UnsupportedFileSystemException - If the file system with specified is not supported

RuntimeException - If the file system specified is supported but could not be instantiated, or if login fails.

getFileContext

public static FileContext getFileContext(Configuration aConf)
                                  throws UnsupportedFileSystemException

Create a FileContext using the passed config. Generally it is better to use getFileContext(URI, Configuration) instead of this one.

Parameters:

aConf -

Returns:

new FileContext

Throws:

UnsupportedFileSystemException - If file system in the config is not supported

getLocalFSFileContext

public static FileContext getLocalFSFileContext(Configuration aConf)
                                         throws UnsupportedFileSystemException

Parameters:

aConf - - from which the FileContext is configured

Returns:

a FileContext for the local file system using the specified config.

Throws:

UnsupportedFileSystemException - If default file system in the config is not supported

setWorkingDirectory

public void setWorkingDirectory(Path newWDir)
                         throws IOException

Set the working directory for wd-relative names (such a "foo/bar"). Working directory feature is provided by simply prefixing relative names with the working dir. Note this is different from Unix where the wd is actually set to the inode. Hence setWorkingDir does not follow symlinks etc. This works better in a distributed environment that has multiple independent roots. getWorkingDirectory() should return what setWorkingDir() set.

Parameters:

newWDir - new working directory

Throws:

IOException -
NewWdir can be one of:

• relative path: "foo/bar";
• absolute without scheme: "/foo/bar"
• fully qualified with scheme: "xx://auth/foo/bar"

Illegal WDs:

• relative with scheme: "xx:foo/bar"
• non existent directory

getWorkingDirectory

public Path getWorkingDirectory()

Gets the working directory for wd-relative names (such a "foo/bar").

getUgi

public UserGroupInformation getUgi()

Gets the ugi in the file-context

Returns:

UserGroupInformation

getHomeDirectory

public Path getHomeDirectory()

Return the current user's home directory in this file system. The default implementation returns "/user/$USER/".

Returns:

the home directory

getUMask

public FsPermission getUMask()

Returns:

the umask of this FileContext

setUMask

public void setUMask(FsPermission newUmask)

Set umask to the supplied parameter.

Parameters:

newUmask - the new umask

resolvePath

public Path resolvePath(Path f)
                 throws FileNotFoundException,
                        org.apache.hadoop.fs.UnresolvedLinkException,
                        AccessControlException,
                        IOException

Resolve the path following any symlinks or mount points

Parameters:

f - to be resolved

Returns:

fully qualified resolved path

Throws:

FileNotFoundException - If f does not exist

AccessControlException - if access denied

IOException - If an IO Error occurred Exceptions applicable to file systems accessed over RPC:

org.apache.hadoop.ipc.RpcClientException - If an exception occurred in the RPC client

org.apache.hadoop.ipc.RpcServerException - If an exception occurred in the RPC server

org.apache.hadoop.ipc.UnexpectedServerException - If server implementation throws undeclared exception to RPC server RuntimeExceptions:

InvalidPathException - If path f is not valid

org.apache.hadoop.fs.UnresolvedLinkException

makeQualified

public Path makeQualified(Path path)

Make the path fully qualified if it is isn't. A Fully-qualified path has scheme and authority specified and an absolute path. Use the default file system and working dir in this FileContext to qualify.

Parameters:

path -

Returns:

qualified path

create

public FSDataOutputStream create(Path f,
                                 EnumSet<CreateFlag> createFlag,
                                 org.apache.hadoop.fs.Options.CreateOpts... opts)
                          throws AccessControlException,
                                 FileAlreadyExistsException,
                                 FileNotFoundException,
                                 ParentNotDirectoryException,
                                 UnsupportedFileSystemException,
                                 IOException

Create or overwrite file on indicated path and returns an output stream for writing into the file.

Parameters:

f - the file name to open

createFlag - gives the semantics of create; see CreateFlag

opts - file creation options; see Options.CreateOpts.

• Progress - to report progress on the operation - default null
• Permission - umask is applied against permission: default is FsPermissions:getDefault()
• CreateParent - create missing parent path; default is to not to create parents
• The defaults for the following are SS defaults of the file server implementing the target path. Not all parameters make sense for all kinds of file system - eg. localFS ignores Blocksize, replication, checksum
  • BufferSize - buffersize used in FSDataOutputStream
  • Blocksize - block size for file blocks
  • ReplicationFactor - replication for blocks
  • ChecksumParam - Checksum parameters. server default is used if not specified.

Returns:

FSDataOutputStream for created file

Throws:

AccessControlException - If access is denied

FileAlreadyExistsException - If file f already exists

FileNotFoundException - If parent of f does not exist and createParent is false

ParentNotDirectoryException - If parent of f is not a directory.

UnsupportedFileSystemException - If file system for f is not supported

IOException - If an I/O error occurred Exceptions applicable to file systems accessed over RPC:

org.apache.hadoop.ipc.RpcClientException - If an exception occurred in the RPC client

org.apache.hadoop.ipc.RpcServerException - If an exception occurred in the RPC server

org.apache.hadoop.ipc.UnexpectedServerException - If server implementation throws undeclared exception to RPC server RuntimeExceptions:

InvalidPathException - If path f is not valid

create

public FSDataOutputStreamBuilder<FSDataOutputStream,?> create(Path f)
                                                       throws IOException

Create a FSDataOutputStreamBuilder for creating or overwriting a file on indicated path.

Parameters:

f - the file path to create builder for.

Returns:

FSDataOutputStreamBuilder to build a FSDataOutputStream. Upon FSDataOutputStreamBuilder.build() being invoked, builder parameters will be verified by FileContext and AbstractFileSystem.create(org.apache.hadoop.fs.Path, java.util.EnumSet<org.apache.hadoop.fs.CreateFlag>, org.apache.hadoop.fs.Options.CreateOpts...). And filesystem states will be modified. Client should expect FSDataOutputStreamBuilder.build() throw the same exceptions as create(Path, EnumSet, CreateOpts...).

Throws:

IOException

mkdir

public void mkdir(Path dir,
                  FsPermission permission,
                  boolean createParent)
           throws AccessControlException,
                  FileAlreadyExistsException,
                  FileNotFoundException,
                  ParentNotDirectoryException,
                  UnsupportedFileSystemException,
                  IOException

Make(create) a directory and all the non-existent parents.

Parameters:

dir - - the dir to make

permission - - permissions is set permission&~umask

createParent - - if true then missing parent dirs are created if false then parent must exist

Throws:

AccessControlException - If access is denied

FileAlreadyExistsException - If directory dir already exists

FileNotFoundException - If parent of dir does not exist and createParent is false

ParentNotDirectoryException - If parent of dir is not a directory

UnsupportedFileSystemException - If file system for dir is not supported

IOException - If an I/O error occurred Exceptions applicable to file systems accessed over RPC:

org.apache.hadoop.ipc.RpcClientException - If an exception occurred in the RPC client

org.apache.hadoop.ipc.UnexpectedServerException - If server implementation throws undeclared exception to RPC server RuntimeExceptions:

InvalidPathException - If path dir is not valid

delete

public boolean delete(Path f,
                      boolean recursive)
               throws AccessControlException,
                      FileNotFoundException,
                      UnsupportedFileSystemException,
                      IOException

Delete a file.

Parameters:

f - the path to delete.

recursive - if path is a directory and set to true, the directory is deleted else throws an exception. In case of a file the recursive can be set to either true or false.

Throws:

AccessControlException - If access is denied

FileNotFoundException - If f does not exist

UnsupportedFileSystemException - If file system for f is not supported

IOException - If an I/O error occurred Exceptions applicable to file systems accessed over RPC:

org.apache.hadoop.ipc.RpcClientException - If an exception occurred in the RPC client

org.apache.hadoop.ipc.RpcServerException - If an exception occurred in the RPC server

org.apache.hadoop.ipc.UnexpectedServerException - If server implementation throws undeclared exception to RPC server RuntimeExceptions:

InvalidPathException - If path f is invalid

open

public FSDataInputStream open(Path f)
                       throws AccessControlException,
                              FileNotFoundException,
                              UnsupportedFileSystemException,
                              IOException

Opens an FSDataInputStream at the indicated Path using default buffersize.

Parameters:

f - the file name to open

Throws:

AccessControlException - If access is denied

FileNotFoundException - If file f does not exist

UnsupportedFileSystemException - If file system for f is not supported

IOException - If an I/O error occurred Exceptions applicable to file systems accessed over RPC:

org.apache.hadoop.ipc.RpcClientException - If an exception occurred in the RPC client

org.apache.hadoop.ipc.RpcServerException - If an exception occurred in the RPC server

org.apache.hadoop.ipc.UnexpectedServerException - If server implementation throws undeclared exception to RPC server

open

public FSDataInputStream open(Path f,
                              int bufferSize)
                       throws AccessControlException,
                              FileNotFoundException,
                              UnsupportedFileSystemException,
                              IOException

Opens an FSDataInputStream at the indicated Path.

Parameters:

f - the file name to open

bufferSize - the size of the buffer to be used.

Throws:

AccessControlException - If access is denied

FileNotFoundException - If file f does not exist

UnsupportedFileSystemException - If file system for f is not supported

IOException - If an I/O error occurred Exceptions applicable to file systems accessed over RPC:

org.apache.hadoop.ipc.RpcClientException - If an exception occurred in the RPC client

org.apache.hadoop.ipc.RpcServerException - If an exception occurred in the RPC server

org.apache.hadoop.ipc.UnexpectedServerException - If server implementation throws undeclared exception to RPC server

truncate

public boolean truncate(Path f,
                        long newLength)
                 throws AccessControlException,
                        FileNotFoundException,
                        UnsupportedFileSystemException,
                        IOException

Truncate the file in the indicated path to the indicated size.

• Fails if path is a directory.
• Fails if path does not exist.
• Fails if path is not closed.
• Fails if new size is greater than current size.

Parameters:

f - The path to the file to be truncated

newLength - The size the file is to be truncated to

Returns:

true if the file has been truncated to the desired newLength and is immediately available to be reused for write operations such as append, or false if a background process of adjusting the length of the last block has been started, and clients should wait for it to complete before proceeding with further file updates.

Throws:

AccessControlException - If access is denied

FileNotFoundException - If file f does not exist

UnsupportedFileSystemException - If file system for f is not supported

IOException - If an I/O error occurred Exceptions applicable to file systems accessed over RPC:

org.apache.hadoop.ipc.RpcClientException - If an exception occurred in the RPC client

org.apache.hadoop.ipc.RpcServerException - If an exception occurred in the RPC server

org.apache.hadoop.ipc.UnexpectedServerException - If server implementation throws undeclared exception to RPC server

setReplication

public boolean setReplication(Path f,
                              short replication)
                       throws AccessControlException,
                              FileNotFoundException,
                              IOException

Set replication for an existing file.

Parameters:

f - file name

replication - new replication

Returns:

true if successful

Throws:

AccessControlException - If access is denied

FileNotFoundException - If file f does not exist

IOException - If an I/O error occurred Exceptions applicable to file systems accessed over RPC:

org.apache.hadoop.ipc.RpcClientException - If an exception occurred in the RPC client

org.apache.hadoop.ipc.RpcServerException - If an exception occurred in the RPC server

org.apache.hadoop.ipc.UnexpectedServerException - If server implementation throws undeclared exception to RPC server

rename

public void rename(Path src,
                   Path dst,
                   org.apache.hadoop.fs.Options.Rename... options)
            throws AccessControlException,
                   FileAlreadyExistsException,
                   FileNotFoundException,
                   ParentNotDirectoryException,
                   UnsupportedFileSystemException,
                   IOException

Renames Path src to Path dst

• Fails if src is a file and dst is a directory.
• Fails if src is a directory and dst is a file.
• Fails if the parent of dst does not exist or is a file.

If OVERWRITE option is not passed as an argument, rename fails if the dst already exists.

If OVERWRITE option is passed as an argument, rename overwrites the dst if it is a file or an empty directory. Rename fails if dst is a non-empty directory.

Note that atomicity of rename is dependent on the file system implementation. Please refer to the file system documentation for details

Parameters:

src - path to be renamed

dst - new path after rename

Throws:

AccessControlException - If access is denied

FileAlreadyExistsException - If dst already exists and options has Options.Rename.OVERWRITE option false.

FileNotFoundException - If src does not exist

ParentNotDirectoryException - If parent of dst is not a directory

UnsupportedFileSystemException - If file system for src and dst is not supported

IOException - If an I/O error occurred Exceptions applicable to file systems accessed over RPC:

org.apache.hadoop.ipc.RpcClientException - If an exception occurred in the RPC client

org.apache.hadoop.ipc.RpcServerException - If an exception occurred in the RPC server

org.apache.hadoop.ipc.UnexpectedServerException - If server implementation throws undeclared exception to RPC server

setPermission

public void setPermission(Path f,
                          FsPermission permission)
                   throws AccessControlException,
                          FileNotFoundException,
                          UnsupportedFileSystemException,
                          IOException

Set permission of a path.

Parameters:

f -

permission - - the new absolute permission (umask is not applied)

Throws:

AccessControlException - If access is denied

FileNotFoundException - If f does not exist

UnsupportedFileSystemException - If file system for f is not supported

IOException - If an I/O error occurred Exceptions applicable to file systems accessed over RPC:

org.apache.hadoop.ipc.RpcClientException - If an exception occurred in the RPC client

org.apache.hadoop.ipc.RpcServerException - If an exception occurred in the RPC server

org.apache.hadoop.ipc.UnexpectedServerException - If server implementation throws undeclared exception to RPC server

setOwner

public void setOwner(Path f,
                     String username,
                     String groupname)
              throws AccessControlException,
                     UnsupportedFileSystemException,
                     FileNotFoundException,
                     IOException

Set owner of a path (i.e. a file or a directory). The parameters username and groupname cannot both be null.

Parameters:

f - The path

username - If it is null, the original username remains unchanged.

groupname - If it is null, the original groupname remains unchanged.

Throws:

AccessControlException - If access is denied

FileNotFoundException - If f does not exist

UnsupportedFileSystemException - If file system for f is not supported

IOException - If an I/O error occurred Exceptions applicable to file systems accessed over RPC:

org.apache.hadoop.ipc.RpcClientException - If an exception occurred in the RPC client

org.apache.hadoop.ipc.RpcServerException - If an exception occurred in the RPC server

org.apache.hadoop.ipc.UnexpectedServerException - If server implementation throws undeclared exception to RPC server RuntimeExceptions:

HadoopIllegalArgumentException - If username or groupname is invalid.

setTimes

public void setTimes(Path f,
                     long mtime,
                     long atime)
              throws AccessControlException,
                     FileNotFoundException,
                     UnsupportedFileSystemException,
                     IOException

Set access time of a file.

Parameters:

f - The path

mtime - Set the modification time of this file. The number of milliseconds since epoch (Jan 1, 1970). A value of -1 means that this call should not set modification time.

atime - Set the access time of this file. The number of milliseconds since Jan 1, 1970. A value of -1 means that this call should not set access time.

Throws:

AccessControlException - If access is denied

FileNotFoundException - If f does not exist

UnsupportedFileSystemException - If file system for f is not supported

IOException - If an I/O error occurred Exceptions applicable to file systems accessed over RPC:

org.apache.hadoop.ipc.RpcClientException - If an exception occurred in the RPC client

org.apache.hadoop.ipc.RpcServerException - If an exception occurred in the RPC server

org.apache.hadoop.ipc.UnexpectedServerException - If server implementation throws undeclared exception to RPC server

getFileChecksum

public FileChecksum getFileChecksum(Path f)
                             throws AccessControlException,
                                    FileNotFoundException,
                                    IOException

Get the checksum of a file.

Parameters:

f - file path

Returns:

The file checksum. The default return value is null, which indicates that no checksum algorithm is implemented in the corresponding FileSystem.

Throws:

AccessControlException - If access is denied

FileNotFoundException - If f does not exist

IOException - If an I/O error occurred Exceptions applicable to file systems accessed over RPC:

org.apache.hadoop.ipc.RpcClientException - If an exception occurred in the RPC client

org.apache.hadoop.ipc.RpcServerException - If an exception occurred in the RPC server

org.apache.hadoop.ipc.UnexpectedServerException - If server implementation throws undeclared exception to RPC server

setVerifyChecksum

public void setVerifyChecksum(boolean verifyChecksum,
                              Path f)
                       throws AccessControlException,
                              FileNotFoundException,
                              UnsupportedFileSystemException,
                              IOException

Set the verify checksum flag for the file system denoted by the path. This is only applicable if the corresponding FileSystem supports checksum. By default doesn't do anything.

Parameters:

verifyChecksum -

f - set the verifyChecksum for the Filesystem containing this path

Throws:

AccessControlException - If access is denied

FileNotFoundException - If f does not exist

UnsupportedFileSystemException - If file system for f is not supported

IOException - If an I/O error occurred Exceptions applicable to file systems accessed over RPC:

org.apache.hadoop.ipc.RpcClientException - If an exception occurred in the RPC client

org.apache.hadoop.ipc.RpcServerException - If an exception occurred in the RPC server

org.apache.hadoop.ipc.UnexpectedServerException - If server implementation throws undeclared exception to RPC server

getFileStatus

public FileStatus getFileStatus(Path f)
                         throws AccessControlException,
                                FileNotFoundException,
                                UnsupportedFileSystemException,
                                IOException

Return a file status object that represents the path.

Parameters:

f - The path we want information from

Returns:

a FileStatus object

Throws:

AccessControlException - If access is denied

FileNotFoundException - If f does not exist

UnsupportedFileSystemException - If file system for f is not supported

IOException - If an I/O error occurred Exceptions applicable to file systems accessed over RPC:

org.apache.hadoop.ipc.RpcClientException - If an exception occurred in the RPC client

org.apache.hadoop.ipc.RpcServerException - If an exception occurred in the RPC server

org.apache.hadoop.ipc.UnexpectedServerException - If server implementation throws undeclared exception to RPC server

getFileLinkStatus

public FileStatus getFileLinkStatus(Path f)
                             throws AccessControlException,
                                    FileNotFoundException,
                                    UnsupportedFileSystemException,
                                    IOException

Return a file status object that represents the path. If the path refers to a symlink then the FileStatus of the symlink is returned. The behavior is equivalent to #getFileStatus() if the underlying file system does not support symbolic links.

Parameters:

f - The path we want information from.

Returns:

A FileStatus object

Throws:

AccessControlException - If access is denied

FileNotFoundException - If f does not exist

UnsupportedFileSystemException - If file system for f is not supported

IOException - If an I/O error occurred

getLinkTarget

public Path getLinkTarget(Path f)
                   throws AccessControlException,
                          FileNotFoundException,
                          UnsupportedFileSystemException,
                          IOException

Returns the target of the given symbolic link as it was specified when the link was created. Links in the path leading up to the final path component are resolved transparently.

Parameters:

f - the path to return the target of

Returns:

The un-interpreted target of the symbolic link.

Throws:

AccessControlException - If access is denied

FileNotFoundException - If path f does not exist

UnsupportedFileSystemException - If file system for f is not supported

IOException - If the given path does not refer to a symlink or an I/O error occurred

getFsStatus

public FsStatus getFsStatus(Path f)
                     throws AccessControlException,
                            FileNotFoundException,
                            UnsupportedFileSystemException,
                            IOException

Returns a status object describing the use and capacity of the file system denoted by the Parh argument p. If the file system has multiple partitions, the use and capacity of the partition pointed to by the specified path is reflected.

Parameters:

f - Path for which status should be obtained. null means the root partition of the default file system.

Returns:

a FsStatus object

Throws:

AccessControlException - If access is denied

FileNotFoundException - If f does not exist

UnsupportedFileSystemException - If file system for f is not supported

IOException - If an I/O error occurred Exceptions applicable to file systems accessed over RPC:

org.apache.hadoop.ipc.RpcClientException - If an exception occurred in the RPC client

org.apache.hadoop.ipc.RpcServerException - If an exception occurred in the RPC server

org.apache.hadoop.ipc.UnexpectedServerException - If server implementation throws undeclared exception to RPC server

createSymlink

public void createSymlink(Path target,
                          Path link,
                          boolean createParent)
                   throws AccessControlException,
                          FileAlreadyExistsException,
                          FileNotFoundException,
                          ParentNotDirectoryException,
                          UnsupportedFileSystemException,
                          IOException

Creates a symbolic link to an existing file. An exception is thrown if the symlink exits, the user does not have permission to create symlink, or the underlying file system does not support symlinks. Symlink permissions are ignored, access to a symlink is determined by the permissions of the symlink target. Symlinks in paths leading up to the final path component are resolved transparently. If the final path component refers to a symlink some functions operate on the symlink itself, these are: - delete(f) and deleteOnExit(f) - Deletes the symlink. - rename(src, dst) - If src refers to a symlink, the symlink is renamed. If dst refers to a symlink, the symlink is over-written. - getLinkTarget(f) - Returns the target of the symlink. - getFileLinkStatus(f) - Returns a FileStatus object describing the symlink. Some functions, create() and mkdir(), expect the final path component does not exist. If they are given a path that refers to a symlink that does exist they behave as if the path referred to an existing file or directory. All other functions fully resolve, ie follow, the symlink. These are: open, setReplication, setOwner, setTimes, setWorkingDirectory, setPermission, getFileChecksum, setVerifyChecksum, getFileBlockLocations, getFsStatus, getFileStatus, exists, and listStatus. Symlink targets are stored as given to createSymlink, assuming the underlying file system is capable of storing a fully qualified URI. Dangling symlinks are permitted. FileContext supports four types of symlink targets, and resolves them as follows

 Given a path referring to a symlink of form:
 
    <---X --->
   fs://host/A/B/link 
    <-----Y ----->
 
 In this path X is the scheme and authority that identify the file system,
 and Y is the path leading up to the final path component "link". If Y is
 a symlink  itself then let Y' be the target of Y and X' be the scheme and
 authority of Y'. Symlink targets may:
 
 1. Fully qualified URIs
 
 fs://hostX/A/B/file  Resolved according to the target file system.
 
 2. Partially qualified URIs (eg scheme but no host)
 
 fs:///A/B/file  Resolved according to the target file system. Eg resolving
                 a symlink to hdfs:///A results in an exception because
                 HDFS URIs must be fully qualified, while a symlink to 
                 file:///A will not since Hadoop's local file systems 
                 require partially qualified URIs.
 
 3. Relative paths
 
 path  Resolves to [Y'][path]. Eg if Y resolves to hdfs://host/A and path 
       is "../B/file" then [Y'][path] is hdfs://host/B/file
 
 4. Absolute paths
 
 path  Resolves to [X'][path]. Eg if Y resolves hdfs://host/A/B and path
       is "/file" then [X][path] is hdfs://host/file
 

Parameters:

target - the target of the symbolic link

link - the path to be created that points to target

createParent - if true then missing parent dirs are created if false then parent must exist

Throws:

AccessControlException - If access is denied

FileAlreadyExistsException - If file link already exists

FileNotFoundException - If target does not exist

ParentNotDirectoryException - If parent of link is not a directory.

UnsupportedFileSystemException - If file system for target or link is not supported

IOException - If an I/O error occurred

listStatus

public org.apache.hadoop.fs.RemoteIterator<FileStatus> listStatus(Path f)
                                                           throws AccessControlException,
                                                                  FileNotFoundException,
                                                                  UnsupportedFileSystemException,
                                                                  IOException

List the statuses of the files/directories in the given path if the path is a directory.

Parameters:

f - is the path

Returns:

an iterator that traverses statuses of the files/directories in the given path

Throws:

AccessControlException - If access is denied

FileNotFoundException - If f does not exist

UnsupportedFileSystemException - If file system for f is not supported

IOException - If an I/O error occurred Exceptions applicable to file systems accessed over RPC:

org.apache.hadoop.ipc.RpcClientException - If an exception occurred in the RPC client

org.apache.hadoop.ipc.RpcServerException - If an exception occurred in the RPC server

org.apache.hadoop.ipc.UnexpectedServerException - If server implementation throws undeclared exception to RPC server

listCorruptFileBlocks

public org.apache.hadoop.fs.RemoteIterator<Path> listCorruptFileBlocks(Path path)
                                                                throws IOException

Returns:

an iterator over the corrupt files under the given path (may contain duplicates if a file has more than one corrupt block)

Throws:

IOException

listLocatedStatus

public org.apache.hadoop.fs.RemoteIterator<LocatedFileStatus> listLocatedStatus(Path f)
                                                                         throws AccessControlException,
                                                                                FileNotFoundException,
                                                                                UnsupportedFileSystemException,
                                                                                IOException

List the statuses of the files/directories in the given path if the path is a directory. Return the file's status and block locations If the path is a file. If a returned status is a file, it contains the file's block locations.

Parameters:

f - is the path

Returns:

an iterator that traverses statuses of the files/directories in the given path If any IO exception (for example the input directory gets deleted while listing is being executed), next() or hasNext() of the returned iterator may throw a RuntimeException with the io exception as the cause.

Throws:

AccessControlException - If access is denied

FileNotFoundException - If f does not exist

UnsupportedFileSystemException - If file system for f is not supported

IOException - If an I/O error occurred Exceptions applicable to file systems accessed over RPC:

org.apache.hadoop.ipc.RpcClientException - If an exception occurred in the RPC client

org.apache.hadoop.ipc.RpcServerException - If an exception occurred in the RPC server

org.apache.hadoop.ipc.UnexpectedServerException - If server implementation throws undeclared exception to RPC server

deleteOnExit

public boolean deleteOnExit(Path f)
                     throws AccessControlException,
                            IOException

Mark a path to be deleted on JVM shutdown.

Parameters:

f - the existing path to delete.

Returns:

true if deleteOnExit is successful, otherwise false.

Throws:

AccessControlException - If access is denied

UnsupportedFileSystemException - If file system for f is not supported

IOException - If an I/O error occurred Exceptions applicable to file systems accessed over RPC:

org.apache.hadoop.ipc.RpcClientException - If an exception occurred in the RPC client

org.apache.hadoop.ipc.RpcServerException - If an exception occurred in the RPC server

org.apache.hadoop.ipc.UnexpectedServerException - If server implementation throws undeclared exception to RPC server

util

public org.apache.hadoop.fs.FileContext.Util util()

resolve

protected Path resolve(Path f)
                throws FileNotFoundException,
                       org.apache.hadoop.fs.UnresolvedLinkException,
                       AccessControlException,
                       IOException

Resolves all symbolic links in the specified path. Returns the new path object.

Throws:

FileNotFoundException

org.apache.hadoop.fs.UnresolvedLinkException

AccessControlException

IOException

resolveIntermediate

protected Path resolveIntermediate(Path f)
                            throws IOException

Resolves all symbolic links in the specified path leading up to, but not including the final path component.

Parameters:

f - path to resolve

Returns:

the new path object.

Throws:

IOException

getStatistics

public static org.apache.hadoop.fs.FileSystem.Statistics getStatistics(URI uri)

Get the statistics for a particular file system

Parameters:

uri - the uri to lookup the statistics. Only scheme and authority part of the uri are used as the key to store and lookup.

Returns:

a statistics object

clearStatistics

public static void clearStatistics()

Clears all the statistics stored in AbstractFileSystem, for all the file systems.

printStatistics

public static void printStatistics()

Prints the statistics to standard output. File System is identified by the scheme and authority.

getAllStatistics

public static Map<URI,org.apache.hadoop.fs.FileSystem.Statistics> getAllStatistics()

Returns:

Map of uri and statistics for each filesystem instantiated. The uri consists of scheme and authority for the filesystem.

modifyAclEntries

public void modifyAclEntries(Path path,
                             List<AclEntry> aclSpec)
                      throws IOException

Modifies ACL entries of files and directories. This method can add new ACL entries or modify the permissions on existing ACL entries. All existing ACL entries that are not specified in this call are retained without changes. (Modifications are merged into the current ACL.)

Parameters:

path - Path to modify

aclSpec - List<AclEntry> describing modifications

Throws:

IOException - if an ACL could not be modified

removeAclEntries

public void removeAclEntries(Path path,
                             List<AclEntry> aclSpec)
                      throws IOException

Removes ACL entries from files and directories. Other ACL entries are retained.

Parameters:

path - Path to modify

aclSpec - List<AclEntry> describing entries to remove

Throws:

IOException - if an ACL could not be modified

removeDefaultAcl

public void removeDefaultAcl(Path path)
                      throws IOException

Removes all default ACL entries from files and directories.

Parameters:

path - Path to modify

Throws:

IOException - if an ACL could not be modified

removeAcl

public void removeAcl(Path path)
               throws IOException

Removes all but the base ACL entries of files and directories. The entries for user, group, and others are retained for compatibility with permission bits.

Parameters:

path - Path to modify

Throws:

IOException - if an ACL could not be removed

setAcl

public void setAcl(Path path,
                   List<AclEntry> aclSpec)
            throws IOException

Fully replaces ACL of files and directories, discarding all existing entries.

Parameters:

path - Path to modify

aclSpec - List<AclEntry> describing modifications, must include entries for user, group, and others for compatibility with permission bits.

Throws:

IOException - if an ACL could not be modified

getAclStatus

public AclStatus getAclStatus(Path path)
                       throws IOException

Gets the ACLs of files and directories.

Parameters:

path - Path to get

Returns:

RemoteIterator<AclStatus> which returns each AclStatus

Throws:

IOException - if an ACL could not be read

setXAttr

public void setXAttr(Path path,
                     String name,
                     byte[] value)
              throws IOException

Set an xattr of a file or directory. The name must be prefixed with the namespace followed by ".". For example, "user.attr".

Refer to the HDFS extended attributes user documentation for details.

Parameters:

path - Path to modify

name - xattr name.

value - xattr value.

Throws:

IOException

setXAttr

public void setXAttr(Path path,
                     String name,
                     byte[] value,
                     EnumSet<XAttrSetFlag> flag)
              throws IOException

Set an xattr of a file or directory. The name must be prefixed with the namespace followed by ".". For example, "user.attr".

Refer to the HDFS extended attributes user documentation for details.

Parameters:

path - Path to modify

name - xattr name.

value - xattr value.

flag - xattr set flag

Throws:

IOException

getXAttr

public byte[] getXAttr(Path path,
                       String name)
                throws IOException

Get an xattr for a file or directory. The name must be prefixed with the namespace followed by ".". For example, "user.attr".

Refer to the HDFS extended attributes user documentation for details.

Parameters:

path - Path to get extended attribute

name - xattr name.

Returns:

byte[] xattr value.

Throws:

IOException

getXAttrs

public Map<String,byte[]> getXAttrs(Path path)
                             throws IOException

Get all of the xattrs for a file or directory. Only those xattrs for which the logged-in user has permissions to view are returned.

Refer to the HDFS extended attributes user documentation for details.

Parameters:

path - Path to get extended attributes

Returns:

Map<String, byte[]> describing the XAttrs of the file or directory

Throws:

IOException

getXAttrs

public Map<String,byte[]> getXAttrs(Path path,
                                    List<String> names)
                             throws IOException

Get all of the xattrs for a file or directory. Only those xattrs for which the logged-in user has permissions to view are returned.

Refer to the HDFS extended attributes user documentation for details.

Parameters:

path - Path to get extended attributes

names - XAttr names.

Returns:

Map<String, byte[]> describing the XAttrs of the file or directory

Throws:

IOException

removeXAttr

public void removeXAttr(Path path,
                        String name)
                 throws IOException

Remove an xattr of a file or directory. The name must be prefixed with the namespace followed by ".". For example, "user.attr".

Refer to the HDFS extended attributes user documentation for details.

Parameters:

path - Path to remove extended attribute

name - xattr name

Throws:

IOException

listXAttrs

public List<String> listXAttrs(Path path)
                        throws IOException

Get all of the xattr names for a file or directory. Only those xattr names which the logged-in user has permissions to view are returned.

Refer to the HDFS extended attributes user documentation for details.

Parameters:

path - Path to get extended attributes

Returns:

List<String> of the XAttr names of the file or directory

Throws:

IOException

createSnapshot

public final Path createSnapshot(Path path)
                          throws IOException

Create a snapshot with a default name.

Parameters:

path - The directory where snapshots will be taken.

Returns:

the snapshot path.

Throws:

IOException - If an I/O error occurred

Exceptions applicable to file systems accessed over RPC:

org.apache.hadoop.ipc.RpcClientException - If an exception occurred in the RPC client

org.apache.hadoop.ipc.RpcServerException - If an exception occurred in the RPC server

org.apache.hadoop.ipc.UnexpectedServerException - If server implementation throws undeclared exception to RPC server

createSnapshot

public Path createSnapshot(Path path,
                           String snapshotName)
                    throws IOException

Create a snapshot.

Parameters:

path - The directory where snapshots will be taken.

snapshotName - The name of the snapshot

Returns:

the snapshot path.

Throws:

IOException - If an I/O error occurred

Exceptions applicable to file systems accessed over RPC:

org.apache.hadoop.ipc.RpcClientException - If an exception occurred in the RPC client

org.apache.hadoop.ipc.RpcServerException - If an exception occurred in the RPC server

org.apache.hadoop.ipc.UnexpectedServerException - If server implementation throws undeclared exception to RPC server

renameSnapshot

public void renameSnapshot(Path path,
                           String snapshotOldName,
                           String snapshotNewName)
                    throws IOException

Rename a snapshot.

Parameters:

path - The directory path where the snapshot was taken

snapshotOldName - Old name of the snapshot

snapshotNewName - New name of the snapshot

Throws:

IOException - If an I/O error occurred

Exceptions applicable to file systems accessed over RPC:

org.apache.hadoop.ipc.RpcClientException - If an exception occurred in the RPC client

org.apache.hadoop.ipc.RpcServerException - If an exception occurred in the RPC server

org.apache.hadoop.ipc.UnexpectedServerException - If server implementation throws undeclared exception to RPC server

deleteSnapshot

public void deleteSnapshot(Path path,
                           String snapshotName)
                    throws IOException

Delete a snapshot of a directory.

Parameters:

path - The directory that the to-be-deleted snapshot belongs to

snapshotName - The name of the snapshot

Throws:

IOException - If an I/O error occurred

Exceptions applicable to file systems accessed over RPC:

org.apache.hadoop.ipc.RpcClientException - If an exception occurred in the RPC client

org.apache.hadoop.ipc.RpcServerException - If an exception occurred in the RPC server

org.apache.hadoop.ipc.UnexpectedServerException - If server implementation throws undeclared exception to RPC server

satisfyStoragePolicy

public void satisfyStoragePolicy(Path path)
                          throws IOException

Set the source path to satisfy storage policy.

Parameters:

path - The source path referring to either a directory or a file.

Throws:

IOException

setStoragePolicy

public void setStoragePolicy(Path path,
                             String policyName)
                      throws IOException

Set the storage policy for a given file or directory.

Parameters:

path - file or directory path.

policyName - the name of the target storage policy. The list of supported Storage policies can be retrieved via getAllStoragePolicies().

Throws:

IOException

unsetStoragePolicy

public void unsetStoragePolicy(Path src)
                        throws IOException

Unset the storage policy set for a given file or directory.

Parameters:

src - file or directory path.

Throws:

IOException

getStoragePolicy

public BlockStoragePolicySpi getStoragePolicy(Path path)
                                       throws IOException

Query the effective storage policy ID for the given file or directory.

Parameters:

path - file or directory path.

Returns:

storage policy for give file.

Throws:

IOException

getAllStoragePolicies

public Collection<? extends BlockStoragePolicySpi> getAllStoragePolicies()
                                                                  throws IOException

Retrieve all the storage policies supported by this file system.

Returns:

all storage policies supported by this filesystem.

Throws:

IOException

openFile

@InterfaceStability.Unstable
public FutureDataInputStreamBuilder openFile(Path path)
                                                                   throws IOException,
                                                                          UnsupportedOperationException

Open a file for reading through a builder API. Ultimately calls open(Path, int) unless a subclass executes the open command differently. The semantics of this call are therefore the same as that of open(Path, int) with one special point: it is in FSDataInputStreamBuilder.build() in which the open operation takes place -it is there where all preconditions to the operation are checked.

Parameters:

path - file path

Returns:

a FSDataInputStreamBuilder object to build the input stream

Throws:

IOException - if some early checks cause IO failures.

UnsupportedOperationException - if support is checked early.