Download Hadoop File System Commands: A Comprehensive List and more Study Guides, Projects, Research Programming Languages in PDF only on Docsity!
Hadoop file system commands
A table of all HDFS operations is reproduced below. The following conventions are used for parameters:
- italics denote variables to be filled out by the user
- “path” means any file or directory name
- “path... ” means one or more file or directory names
- “file” means any filename.
- “src” and “dest” are path names in a directed operation.
- “localSrc” and “localDest” are paths as above, but on the local file system. All other file and path names refer to objects inside HDFS
- parameters in [brackets] are optional
Command: hadoop fs ... Description -ls path Lists the contents of the directory spec- ified by path, showing the names, per- missions, owner, size and modification date for each entry. -ls -R path Behaves like -ls, but recursively displays entries in all subdirectories of path. -du path Shows disk usage, in bytes, for all files which match path; filenames are re- ported with the full HDFS protocol pre- fix. -du -s path Like -du, but prints a summary of disk usage of all files/directories in the path. -mv src dest Moves the file or directory indicated by src to dest, within HDFS. -cp src dest Copies the file or directory identified by src to dest, within HDFS. -rm path Removes the file or empty directory identified by path.
-rm -r path Removes the file or directory identified by path. Recursively deletes any child entries (i.e., files or subdirectories of path).
-put localSrc dest Copies the file or directory from the lo- cal file system identified by localSrc to dest within the DFS.
-copyFromLocal localSrc dest Identical to -put
-moveFromLocal localSrc dest Copies the file or directory from the lo- cal file system identified by localSrc to dest within HDFS, then deletes the lo- cal copy on success.
-get [-crc] src localDest Copies the file or directory in HDFS identified by src to the local file system path identified by localDest.
-getmerge src localDest [addnl] Retrieves all files that match the path src in HDFS, and copies them to a sin- gle, merged file in the local file system identified by localDest.
-cat filename Displays the contents of filename on std- out.
-copyToLocal [-crc] src localDest Identical to -get
-moveToLocal [-crc] src localDest Works like -get, but deletes the HDFS copy on success.
-mkdir path Creates a directory named path in HDFS. Creates any parent directories in path that are missing (e.g., like mkdir -p in Linux).
-setrep [-R] [-w] rep path Sets the target replication factor for files identified by path to rep. (The actual replication factor will move toward the target over time)
-touchz path Creates a file at path containing the cur- rent time as a timestamp. Fails if a file already exists at path, unless the file is already size 0.
-test -[ezd] path Returns 1 if path exists; has zero length; or is a directory, or 0 otherwise.
-stat [format] path Prints information about path. format is a string which accepts file size in blocks (%b), filename (%n), block size (%o), replication (%r), and modification date (%y, %Y).
