apache/hadoop · error · FileAlreadyExistsException

"Cannot append to directory " + path + "; already exists as

Error message

"Cannot append to directory " + path + "; already exists as a directory."

What it means

FSDirAppendOp.appendFile throws FileAlreadyExistsException when the append target resolves to an existing directory. HDFS append only extends an existing file - it never creates or replaces paths - so a directory target is rejected right after resolvePath and before permission checks.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java:98

   * @return the last block with status
   */
  static LastBlockWithStatus appendFile(final FSNamesystem fsn,
      final String srcArg, final FSPermissionChecker pc, final String holder,
      final String clientMachine, final boolean newBlock,
      final boolean logRetryCache) throws IOException {
    assert fsn.hasWriteLock(RwLockMode.GLOBAL);

    final LocatedBlock lb;
    final FSDirectory fsd = fsn.getFSDirectory();
    final INodesInPath iip;
    fsd.writeLock();
    try {
      iip = fsd.resolvePath(pc, srcArg, DirOp.WRITE);
      // Verify that the destination does not exist as a directory already
      final INode inode = iip.getLastINode();
      final String path = iip.getPath();
      if (inode != null && inode.isDirectory()) {
        throw new FileAlreadyExistsException("Cannot append to directory "
            + path + "; already exists as a directory.");
      }
      if (fsd.isPermissionEnabled()) {
        fsd.checkPathAccess(pc, iip, FsAction.WRITE);
      }

      if (inode == null) {
        throw new FileNotFoundException(
            "Failed to append to non-existent file " + path + " for client "
                + clientMachine);
      }
      final INodeFile file = INodeFile.valueOf(inode, path, true);

      if (file.isStriped() && !newBlock) {
        throw new UnsupportedOperationException(
            "Append on EC file without new block is not supported. Use "
                + CreateFlag.NEW_BLOCK + " create flag while appending file.");
      }

View on GitHub (pinned to 2add963021)

Solutions

Check fs.getFileStatus(path).isDirectory() before calling append and fail fast with a clear message.
Point the append at a file under the directory (e.g. /jobs/out/part-0) and make sure no upstream stage mkdirs() that exact file path.
If the directory is stale output from an earlier layout, delete or rename it before writing.

Example fix

// before
FSDataOutputStream out = fs.append(path);

// after
if (fs.exists(path) && fs.getFileStatus(path).isDirectory()) {
  throw new IllegalArgumentException(path + " is a directory; append to a file inside it");
}
FSDataOutputStream out = fs.append(path);

Defensive patterns

Strategy: validation

Validate before calling

static void assertAppendTargetIsFile(FileSystem fs, Path p) throws IOException {
  if (fs.exists(p) && fs.getFileStatus(p).isDirectory()) {
    throw new IllegalArgumentException(p + " is a directory; append to a file path under it");
  }
}

Type guard

static boolean isAppendablePath(FileSystem fs, Path p) throws IOException {
  return !fs.exists(p) || !fs.getFileStatus(p).isDirectory(); // missing files fail later with FNFE
}

Try / catch

try {
  out = fs.append(path);
} catch (FileAlreadyExistsException e) {
  // append hit a directory: fix the path configuration, do not retry the same path
}

Prevention

Never mkdirs() an exact path a later stage appends to; separate dir creation from file paths.
Validate output path templates once at job startup.

When it happens

Trigger: ClientProtocol.append / DistributedFileSystem.append(path) or 'hdfs dfs -appendToFile ... <dst>' where dst is a directory: an output path like /jobs/out that an earlier run created with mkdirs, or a path whose last component exists as a directory.

Common situations: Job drivers that append to a 'current' file whose path the previous stage mkdirs()-ed; path construction bugs pointing at the directory instead of a file inside it; output-committers pre-creating the _temporary target path.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5b68e20e83d2f992. Report an issue: GitHub.