What causes "Parent directory doesn't exist: {}" in apache/hadoop?

Create the parent explicitly before writing: if (!fs.exists(parent)) fs.mkdirs(parent); then call createNonRecursive.

apache/hadoop · error · FileNotFoundException

Parent directory doesn't exist: {}

Error message

Parent directory doesn't exist: {}

What it means

Inside ChecksumFileSystem.create, when called with createParent=false (which is what createNonRecursive does) and the target's parent directory does not exist, the write is refused with FileNotFoundException naming the missing parent. This is a deliberate guard: 'non-recursive' create promises not to silently mkdir parents, so a missing parent is an error, not something to fix.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java:770

    }
  }

  @Override
  public FSDataOutputStream create(Path f, FsPermission permission,
      boolean overwrite, int bufferSize, short replication, long blockSize,
      Progressable progress) throws IOException {
    return create(f, permission, overwrite, true, bufferSize,
        replication, blockSize, progress);
  }

  private FSDataOutputStream create(Path f, FsPermission permission,
      boolean overwrite, boolean createParent, int bufferSize,
      short replication, long blockSize,
      Progressable progress) throws IOException {
    Path parent = f.getParent();
    if (parent != null) {
      if (!createParent && !exists(parent)) {
        throw new FileNotFoundException("Parent directory doesn't exist: "
            + parent);
      } else if (!mkdirs(parent)) {
        throw new IOException("Mkdirs failed to create " + parent
            + " (exists=" + exists(parent) + ", cwd=" + getWorkingDirectory()
            + ")");
      }
    }
    final FSDataOutputStream out;
    if (writeChecksum) {
      out = new FSDataOutputStream(
          new ChecksumFSOutputSummer(this, f, overwrite, bufferSize, replication,
              blockSize, progress, permission), null);
    } else {
      out = fs.create(f, permission, overwrite, bufferSize, replication,
          blockSize, progress);
      // remove the checksum file since we aren't writing one
      Path checkFile = getChecksumFile(f);
      if (fs.exists(checkFile)) {

View on GitHub (pinned to 2add963021)

Solutions

Create the parent explicitly before writing: if (!fs.exists(parent)) fs.mkdirs(parent); then call createNonRecursive.
Switch to plain fs.create(...), which passes createParent=true and auto-mkdirs the parent chain.
Print fs.getWorkingDirectory() and the absolute path (the error only names the parent) to catch relative-path/working-directory mistakes.

Example fix

// before
FSDataOutputStream out = fs.createNonRecursive(new Path("out/part-0"), perms, true, 4096, (short)1, 1<<26, null);
// throws FileNotFoundException: Parent directory doesn't exist: out

// after
Path parent = new Path("out");
if (!fs.exists(parent)) fs.mkdirs(parent);
FSDataOutputStream out = fs.createNonRecursive(new Path(parent, "part-0"), perms, true, 4096, (short)1, 1<<26, null);

Defensive patterns

Strategy: validation

Validate before calling

Path parent = path.getParent();
if (parent != null && !fs.exists(parent)) {
  fs.mkdirs(parent); // or fail with a clear message before createNonRecursive
}

Try / catch

try {
  out = fs.createNonRecursive(path, perms, overwrite, buf, rep, block, null);
} catch (FileNotFoundException e) {
  // e.getMessage() names the missing parent; mkdirs and retry once
}

Prevention

Use plain create() when implicit parent creation is acceptable.
With createNonRecursive, always precede it with an explicit exists/mkdirs on the parent.
Log fs.getWorkingDirectory() alongside relative paths to catch resolution surprises.

When it happens

Trigger: Calling fs.createNonRecursive(path, permission, overwrite, ...) on LocalFileSystem (ChecksumFileSystem.java:796 forwards to the private create with createParent=false) when path.getParent() does not exist. Also reachable via FSDataOutputStreamBuilder / FileSystem.create(..., EnumSet<CreateFlag>, ...) variants that route to createNonRecursive.

Common situations: Job committers and output writers using createNonRecursive to avoid racing implicit mkdirs (classic MR FileOutputCommitter pattern); a parent directory deleted by a concurrent cleanup job; wrong working directory or misspelled relative path making the resolved parent not exist.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4ae4f42c9a1cbf00. Report an issue: GitHub.