What causes "failed to append to non-existent file {} on client {}" in apache/hadoop?

If create-or-append is intended, request both flags, e.g. DistributedFileSystem.create(path, EnumSet.of(CreateFlag.CREATE, CreateFlag.APPEND), ...) — atomically appends or creates.

apache/hadoop · error · FileNotFoundException

failed to append to non-existent file {} on client {}

Error message

failed to append to non-existent file {} on client {}

What it means

When CreateFlag.APPEND is requested, DFSClient.primitiveAppend first stats the target with getFileInfo(); if the file is absent and CreateFlag.CREATE is not also set, append cannot proceed and throws FileNotFoundException naming the src and client. Plain append never creates the file; only APPEND+CREATE has create-or-append semantics.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java:1350

      for (int i = 0; i < favoredNodes.length; i++) {
        favoredNodeStrs[i] =
            favoredNodes[i].getHostName() + ":" + favoredNodes[i].getPort();
      }
    }
    return favoredNodeStrs;
  }

  /**
   * Append to an existing file if {@link CreateFlag#APPEND} is present
   */
  private DFSOutputStream primitiveAppend(String src, EnumSet<CreateFlag> flag,
      Progressable progress) throws IOException {
    if (flag.contains(CreateFlag.APPEND)) {
      HdfsFileStatus stat = getFileInfo(src);
      if (stat == null) { // No file to append to
        // New file needs to be created if create option is present
        if (!flag.contains(CreateFlag.CREATE)) {
          throw new FileNotFoundException(
              "failed to append to non-existent file " + src + " on client "
                  + clientName);
        }
        return null;
      }
      return callAppend(src, flag, progress, null);
    }
    return null;
  }

  /**
   * Same as {{@link #create(String, FsPermission, EnumSet, short, long,
   *  Progressable, int, ChecksumOpt)} except that the permission
   *  is absolute (ie has already been masked with umask.
   */
  public DFSOutputStream primitiveCreate(String src, FsPermission absPermission,
      EnumSet<CreateFlag> flag, boolean createParent, short replication,
      long blockSize, Progressable progress, int buffersize,

View on GitHub (pinned to 2add963021)

Solutions

If create-or-append is intended, request both flags, e.g. DistributedFileSystem.create(path, EnumSet.of(CreateFlag.CREATE, CreateFlag.APPEND), ...) — atomically appends or creates.
Otherwise create the file explicitly on first write (fs.create(path, true)), then append on subsequent writes.
Verify the path computation (rotation index, date suffix) — most 'missing' appends target the wrong name.
Catch FileNotFoundException at the append as the create trigger instead of crashing the writer.

Example fix

// before
try (FSDataOutputStream out = fs.append(path)) { ... }
// FileNotFoundException: failed to append to non-existent file <path> on client <name>

// after: create-or-append (for a race-free variant use
// ((DistributedFileSystem) fs).create(path, EnumSet.of(CreateFlag.CREATE, CreateFlag.APPEND), ...))
try (FSDataOutputStream out = fs.exists(path)
        ? fs.append(path)
        : fs.create(path, true)) {
  // ...
}

Defensive patterns

Strategy: validation

Validate before calling

// create-or-append before writing
if (!fs.exists(path)) {
  out = fs.create(path, true);
} else {
  out = fs.append(path);
}

Try / catch

catch (FileNotFoundException e) {
  out = fs.create(path, true); // first write creates the file, then append
}

Prevention

Design sinks as create-on-first-write, append-afterwards.
Use EnumSet.of(CreateFlag.CREATE, CreateFlag.APPEND) for atomic create-or-append.
Log rotation filenames to catch suffix/rotation bugs early.

When it happens

Trigger: fs.append(path) (or create with EnumSet.of(CreateFlag.APPEND) alone) on a file that was deleted or never written: append-based sinks starting before the first create, reruns of jobs after cleanup deleted outputs, or a rotation suffix computing a filename that does not exist yet.

Common situations: Log-writer/Flume style sinks in append mode whose target file was rolled away or never created; rerun-after-cleanup workflows; date-based filename rotation landing on a new name for the first time.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9d713da642fa96d0. Report an issue: GitHub.