apache/beam · error · FileNotFoundException

Unable to rename resource

Error message

Unable to rename resource %s to %s as source not found.

What it means

When Hadoop's fs.rename() returns false, rename() diagnoses the cause: if the source path no longer exists it throws FileNotFoundException with this message. Hadoop returns false instead of throwing, so Beam translates the boolean into a precise exception.

Solutions

Verify the source path exists (fs.exists / HadoopFileSystem.match) before renaming.
Fix the path/scheme (e.g. hdfs:// vs local) used to construct the resource id.
Add retry/dependency ordering so the rename only runs after the producing step succeeds.

Example fix

// before
fileSystem.rename(ImmutableList.of(src), ImmutableList.of(dest));
// after
MatchResult match = fileSystem.match(src.toString());
if (match.status() == Status.OK) {
  fileSystem.rename(ImmutableList.of(src), ImmutableList.of(dest));
}

Defensive patterns

Strategy: validation

Validate before calling

MatchResult mr = fileSystem.match(src.toString());
if (mr.status() != MatchResult.Status.OK) {
  throw new FileNotFoundException("Source missing: " + src);
}

Try / catch

try {
  fileSystem.rename(srcs, dests);
} catch (FileNotFoundException e) {
  LOG.warn("Rename source missing, skipping: {}", e.getMessage());
}

Prevention

Match/verify source paths before renaming
Order pipeline steps so renames follow successful writes
Use consistent schemes/authorities when building ResourceIds

When it happens

Trigger: Calling HadoopFileSystem.rename() with a srcResourceIds entry whose path does not exist in the underlying Hadoop filesystem, causing fs.rename to return false and fs.exists(srcPath) to be false.

Common situations: Renaming files produced by an upstream pipeline step that failed or was skipped; stale paths after a job rerun; typo'd or wrong-filesystem path.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ab5fc349ed9a7cf9. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystem.java:280

      final org.apache.hadoop.fs.FileSystem fs = srcPath.getFileSystem(configuration);

      // rename in HDFS requires the target directory to exist or silently fails (BEAM-4861)
      mkdirs(destPath);

      boolean success = fs.rename(srcPath, destPath);

      // If the failure was due to the file already existing, delete and retry (BEAM-5036).
      // This should be the exceptional case, so handle here rather than incur the overhead of
      // testing first
      if (!success && fs.exists(srcPath) && fs.exists(destPath)) {
        LOG.debug(LOG_DELETING_EXISTING_FILE, Path.getPathWithoutSchemeAndAuthority(destPath));
        fs.delete(destPath, false); // not recursive
        success = fs.rename(srcPath, destPath);
      }

      if (!success) {
        if (!fs.exists(srcPath)) {
          throw new FileNotFoundException(
              String.format(
                  "Unable to rename resource %s to %s as source not found.", srcPath, destPath));

        } else if (fs.exists(destPath)) {
          throw new FileAlreadyExistsException(
              String.format(
                  "Unable to rename resource %s to %s as destination already exists and couldn't be deleted.",
                  srcPath, destPath));

        } else {
          throw new IOException(
              String.format(
                  "Unable to rename resource %s to %s. No further information provided by underlying filesystem.",
                  srcPath, destPath));
        }
      }
    }
  }

View on GitHub (pinned to 12126d8942)