apache/hadoop · error · FileNotFoundException

File/Directory {path} does not exist

Error message

File/Directory {path} does not exist

What it means

Thrown by IOUtils.fsync(File) when the path to sync does not exist on the local filesystem at the time the channel is opened. It is a plain FileNotFoundException: between the caller's creation of the file and the fsync call, the file was deleted, or it was never created. The sync never runs — this is a precondition check, not a sync failure.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java:394

        }
      }
    } catch (DirectoryIteratorException e) {
      throw e.getCause();
    }
    return list;
  }

  /**
   * Ensure that any writes to the given file is written to the storage device
   * that contains it. This method opens channel on given File and closes it
   * once the sync is done.<br>
   * Borrowed from Uwe Schindler in LUCENE-5588
   * @param fileToSync the file to fsync
   * @throws IOException raised on errors performing I/O.
   */
  public static void fsync(File fileToSync) throws IOException {
    if (!fileToSync.exists()) {
      throw new FileNotFoundException(
          "File/Directory " + fileToSync.getAbsolutePath() + " does not exist");
    }
    boolean isDir = fileToSync.isDirectory();

    // HDFS-13586, FileChannel.open fails with AccessDeniedException
    // for any directory, ignore.
    if (isDir && Shell.WINDOWS) {
      return;
    }

    // If the file is a directory we have to open read-only, for regular files
    // we must open r/w for the fsync to have an effect. See
    // http://blog.httrack.com/blog/2013/11/15/
    // everything-you-always-wanted-to-know-about-fsync/
    try(FileChannel channel = FileChannel.open(fileToSync.toPath(),
        isDir ? StandardOpenOption.READ : StandardOpenOption.WRITE)){
      fsync(channel, isDir);
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Check for concurrent deletion of the file (logs, retention/cleanup threads) and serialize delete vs. sync with a lock or lease.
2. Verify ordering: the file must be created and still open/flushed before fsync is called.
3. Re-create the file and retry the write+fsync if the artifact is recoverable.
4. If the path is wrong, fix the path construction — print fileToSync.getAbsolutePath() from the message and compare with the intended location.

Example fix

// before: delete and sync race; FileNotFoundException under load
if (!file.exists() || !file.isDirectory()) {
  IOUtils.fsync(file);
}

// after: hold the lifecycle lock and re-check existence atomically with sync
synchronized (fileLifecycleLock) {
  if (file.exists()) {
    IOUtils.fsync(file);
  } else {
    LOG.warn("{} vanished before fsync — skipped", file.getAbsolutePath());
  }
}

synchronized (lifecycleLock) {
  if (!fileToSync.exists()) {
    LOG.warn("{} deleted before fsync — skipping", fileToSync);
    return;
  }
  IOUtils.fsync(fileToSync);
}

try {
  IOUtils.fsync(file);
} catch (FileNotFoundException e) {
  // file vanished between creation and sync —
  // re-create and rewrite if recoverable, else skip with a warning
  handleVanishedFile(file);
}

When it happens

Trigger: Calling IOUtils.fsync(fileToSync) on a path that was removed (e.g. a log segment deleted by a concurrent process) or on a path created only later (ordering bug: fsync before create/flush). Note from the source that on Windows, directory fsync returns early, so this fires for directories only on non-Windows platforms.

Common situations: NameNode/edit-log style code where a cleaner thread deletes segments while a writer tries to roll+fsync; checkpoint artifacts deleted by a timeout between write and sync; unit tests on tmp dirs cleaned concurrently; passing a mistyped path that was never written.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• No such file or directory
• Path points to dir not a file
• Unable to write to output stream.
• Error while reading compressed data
• Premature EOF from inputStream

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/fc94b234cd80560b. Report an issue: GitHub.