apache/hadoop · error · FileNotFoundException

Non existing file: ${path}. Create option is not specified i

Error message

Non existing file: ${path}. Create option is not specified in ${flag}

What it means

CreateFlag.validate(path, pathExists=false, flag) throws FileNotFoundException ('Non existing file: ... Create option is not specified') when the target does not exist and the flags lack CREATE. Effectively: you asked to append to (or otherwise touch) a file that is not there and did not authorize creating it.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CreateFlag.java:184

   * @param path Object representing the path; usually String or {@link Path}
   * @param pathExists pass true if the path exists in the file system
   * @param flag set of CreateFlag
   * @throws IOException on error
   * @throws HadoopIllegalArgumentException if the CreateFlag is invalid
   */
  public static void validate(Object path, boolean pathExists,
      EnumSet<CreateFlag> flag) throws IOException {
    validate(flag);
    final boolean append = flag.contains(APPEND);
    final boolean overwrite = flag.contains(OVERWRITE);
    if (pathExists) {
      if (!(append || overwrite)) {
        throw new FileAlreadyExistsException("File already exists: "
            + path.toString()
            + ". Append or overwrite option must be specified in " + flag);
      }
    } else if (!flag.contains(CREATE)) {
      throw new FileNotFoundException("Non existing file: " + path.toString()
          + ". Create option is not specified in " + flag);
    }
  }

  /**
   * Validate the CreateFlag for the append operation. The flag must contain
   * APPEND, and cannot contain OVERWRITE.
   *
   * @param flag enum set flag.
   */
  public static void validateForAppend(EnumSet<CreateFlag> flag) {
    validate(flag);
    if (!flag.contains(APPEND)) {
      throw new HadoopIllegalArgumentException(flag
          + " does not contain APPEND");
    }
  }
}

View on GitHub (pinned to 2add963021)

Solutions

Include CREATE together with APPEND if your filesystem honors create-on-append: EnumSet.of(CreateFlag.CREATE, CreateFlag.APPEND) (HDFS supports this: creates then appends).
Otherwise create explicitly first when missing: if (!fs.exists(f)) { try (out = fs.create(f)) {} } before the append.
Check path spelling and fs.getWorkingDirectory() - the exists()=false result is often a relative-path surprise.

Example fix

// before
EnumSet<CreateFlag> flags = EnumSet.of(CreateFlag.APPEND); // first run, file absent
fs.create(path, perms, flags, 4096, (short)1, 1<<26, null); // FileNotFoundException

// after
EnumSet<CreateFlag> flags = fs.exists(path)
    ? EnumSet.of(CreateFlag.APPEND)
    : EnumSet.of(CreateFlag.CREATE, CreateFlag.APPEND);

Defensive patterns

Strategy: validation

Validate before calling

if (!fs.exists(path) && !flags.contains(CreateFlag.CREATE)) {
  flags = EnumSet.copyOf(flags);
  flags.add(CreateFlag.CREATE); // or fail with a clear message
}

Try / catch

try {
  out = fs.append(path);
} catch (FileNotFoundException e) { // 'Non existing file... Create option is not specified'
  // first run: create the file, then append next time (or use CREATE+APPEND where supported)
}

Prevention

For append-style writers, handle the first run explicitly (create empty file at startup).
Use EnumSet.of(CREATE, APPEND) on filesystems that support create-on-append.
Log the resolved absolute path when the file is reported missing - it is often a cwd issue.

When it happens

Trigger: Calling create/append with EnumSet.of(CreateFlag.APPEND) on a path that does not exist (e.g., first run of a log writer), or any flag set without CREATE against a missing file - validated from fs.create(..., EnumSet, ...), fc.create, and append paths that call validate.

Common situations: Log/offset writers that append on every run but crash or get their first run against a fresh directory; typo'd or wrong-working-directory path resolving to a nonexistent file; a race where another process removed the file between the exists() check and the create call.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/771f9a8e915f5d5a. Report an issue: GitHub.