apache/hadoop · error · IOException

Cannot find source file

Error message

Cannot find source file - {sourceFile}

What it means

Companion check in RouterAsyncClientProtocol.concat: while iterating the source files, each srcLocation is resolved with getFileRemoteLocation; null means that source file could not be found in any namespace and IOException('Cannot find source file - <sourceFile>') aborts the operation. One missing source blocks the whole concat because the operation must atomically merge all listed files into the target.

Solutions

1. Validate every source path exists (hdfs dfs -ls or getFileInfo batch) before calling concat
2. Rebuild the source list from a fresh listing of the target directory at concat time
3. Fix mount entries if sources exist in a subcluster but are not visible through the Router

// Validate all sources exist before concat:
for (String s : srcs) {
  if (dfs.getClient().getNamenode().getFileInfo(s) == null) {
    throw new IllegalArgumentException("missing source: " + s);
  }
}

catch (IOException e) {
  if (e.getMessage().startsWith("Cannot find source file - ")) {
    // identify the missing part file, regenerate/skip it, rebuild src list
  }
}

When it happens

Trigger: concat(trg, srcs) where any entry of srcs does not exist (typo, deleted, or still being written); a source path resolving to a namespace where the file is absent.

Common situations: Job commit concatenating part files when one part was cleaned up or failed to flush; partial uploads; source list built from a stale directory listing.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• Cannot find target file
• Cannot concatenate source file
• File not found.
• No mount point for
• No namespace availaible.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c94db295e16222e8. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/async/RouterAsyncClientProtocol.java:329

  @Override
  public void concat(String trg, String[] src) throws IOException {
    rpcServer.checkOperation(NameNode.OperationCategory.WRITE);

    // Concat only effects when all files in the same namespace.
    getFileRemoteLocation(trg);
    asyncApply((AsyncApplyFunction<RemoteLocation, Object>) targetDestination -> {
      if (targetDestination == null) {
        throw new IOException("Cannot find target file - " + trg);
      }
      String targetNameService = targetDestination.getNameserviceId();
      String[] sourceDestinations = new String[src.length];
      int[] index = new int[1];
      asyncForEach(Arrays.stream(src).iterator(), (forEachRun, sourceFile) -> {
        getFileRemoteLocation(sourceFile);
        asyncApply((ApplyFunction<RemoteLocation, Object>) srcLocation -> {
          if (srcLocation == null) {
            throw new IOException("Cannot find source file - " + sourceFile);
          }
          sourceDestinations[index[0]++] = srcLocation.getDest();
          if (!targetNameService.equals(srcLocation.getNameserviceId())) {
            throw new IOException("Cannot concatenate source file " + sourceFile
                + " because it is located in a different namespace" + " with nameservice "
                + srcLocation.getNameserviceId() + " from the target file with nameservice "
                + targetNameService);
          }
          return null;
        });
      });
      asyncApply((AsyncApplyFunction<Object, Object>) o -> {
        // Invoke
        RemoteMethod method = new RemoteMethod("concat",
            new Class<?>[] {String.class, String[].class},
            targetDestination.getDest(), sourceDestinations);
        rpcClient.invokeSingle(targetDestination, method, Void.class);
      });

View on GitHub (pinned to 2add963021)