What causes "Not a file: %s" in apache/hadoop?

Confirm the staged file still exists and is a regular file before the commit phase (File#isFile check in a precommit hook)

apache/hadoop · error · FileNotFoundException

Not a file: %s

Error message

Not a file: %s

What it means

FileNotFoundException from CommitOperations.uploadFileToPendingCommit when the local file to upload does not exist as a regular file (localFile.isFile() is false). This is the entry point that turns a locally staged task output into a multipart upload plus a SinglePendingCommit record, so the local file must still be present and readable at commit time. The check runs after the 'Initiating multipart upload' debug log and before any S3 call.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/impl/CommitOperations.java:533

   * @param destPath destination path
   * @param partition partition/subdir. Not used
   * @param uploadPartSize size of upload
   * @param progress progress callback
   * @return a pending upload entry
   * @throws IOException failure
   */
  public SinglePendingCommit uploadFileToPendingCommit(File localFile,
      Path destPath,
      String partition,
      long uploadPartSize,
      Progressable progress)
      throws IOException {

    LOG.debug("Initiating multipart upload from {} to {}",
        localFile, destPath);
    Preconditions.checkArgument(destPath != null);
    if (!localFile.isFile()) {
      throw new FileNotFoundException("Not a file: " + localFile);
    }
    String destURI = destPath.toUri().toString();
    String destKey = fs.pathToKey(destPath);
    String uploadId = null;

    // flag to indicate to the finally clause that the operation
    // failed. it is cleared as the last action in the try block.
    boolean threw = true;
    final DurationTracker tracker = statistics.trackDuration(
        COMMITTER_STAGE_FILE_UPLOAD.getSymbol());
    try (DurationInfo d = new DurationInfo(LOG,
        "Upload staged file from %s to %s",
        localFile.getAbsolutePath(),
        destPath)) {

      statistics.commitCreated();
      uploadId = writeOperations.initiateMultiPartUpload(destKey,
          PutObjectOptions.defaultOptions());

View on GitHub (pinned to 2add963021)

Solutions

Confirm the staged file still exists and is a regular file before the commit phase (File#isFile check in a precommit hook)
Keep task attempt directories intact until job commit finishes: check NodeManager disk utilization and local-dirs cleanup policy on the node that ran the task
If files were lost, rerun the failed job rather than trying to hand-craft pending commit records

Example fix

// before
commitData = ops.uploadFileToPendingCommit(localFile, destPath, partition, partSize, progress);

// after
if (!localFile.isFile()) {
  throw new IOException("Staged file missing before commit: " + localFile);
}
commitData = ops.uploadFileToPendingCommit(localFile, destPath, partition, partSize, progress);

Defensive patterns

Strategy: validation

Validate before calling

if (!localFile.isFile() || !localFile.canRead()) {
  throw new IOException("Staged output missing/unreadable before commit: "
      + localFile.getAbsolutePath());
}

Try / catch

try {
  SinglePendingCommit data = ops.uploadFileToPendingCommit(
      localFile, destPath, partition, partSize, progress);
} catch (FileNotFoundException e) {
  // local staging file vanished: fail the task so the attempt reruns on a healthy node
  LOG.error("Staged file lost before upload: {}", localFile, e);
  throw e;
}

Prevention

Monitor NodeManager local-dirs disk usage so eviction cannot delete live attempt directories
Never clean task attempt directories before job commit completes
In precommit hooks, verify every expected staged file exists before letting commitJob start

When it happens

Trigger: uploadFileToPendingCommit is called with a File that was deleted (task attempt working directory cleaned up) or that names a directory rather than a file.

Common situations: NodeManager local directories are cleaned (disk-policy eviction or yarn.nodemanager.local-dirs housekeeping) before job commit; a task retry removed the previous attempt's files; someone manually deleted the staging directory; the path points at the attempt directory itself instead of a file inside it.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8ad7a56c4cc8af98. Report an issue: GitHub.