apache/hadoop · error · IOException

Cannot find source file - {sourceFile}

Error message

Cannot find source file - {sourceFile}

What it means

Companion check in RouterAsyncClientProtocol.concat: while iterating the source files, each srcLocation is resolved with getFileRemoteLocation; null means that source file could not be found in any namespace and IOException('Cannot find source file - <sourceFile>') aborts the operation. One missing source blocks the whole concat because the operation must atomically merge all listed files into the target.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/async/RouterAsyncClientProtocol.java:329

  @Override
  public void concat(String trg, String[] src) throws IOException {
    rpcServer.checkOperation(NameNode.OperationCategory.WRITE);

    // Concat only effects when all files in the same namespace.
    getFileRemoteLocation(trg);
    asyncApply((AsyncApplyFunction<RemoteLocation, Object>) targetDestination -> {
      if (targetDestination == null) {
        throw new IOException("Cannot find target file - " + trg);
      }
      String targetNameService = targetDestination.getNameserviceId();
      String[] sourceDestinations = new String[src.length];
      int[] index = new int[1];
      asyncForEach(Arrays.stream(src).iterator(), (forEachRun, sourceFile) -> {
        getFileRemoteLocation(sourceFile);
        asyncApply((ApplyFunction<RemoteLocation, Object>) srcLocation -> {
          if (srcLocation == null) {
            throw new IOException("Cannot find source file - " + sourceFile);
          }
          sourceDestinations[index[0]++] = srcLocation.getDest();
          if (!targetNameService.equals(srcLocation.getNameserviceId())) {
            throw new IOException("Cannot concatenate source file " + sourceFile
                + " because it is located in a different namespace" + " with nameservice "
                + srcLocation.getNameserviceId() + " from the target file with nameservice "
                + targetNameService);
          }
          return null;
        });
      });
      asyncApply((AsyncApplyFunction<Object, Object>) o -> {
        // Invoke
        RemoteMethod method = new RemoteMethod("concat",
            new Class<?>[] {String.class, String[].class},
            targetDestination.getDest(), sourceDestinations);
        rpcClient.invokeSingle(targetDestination, method, Void.class);
      });

View on GitHub (pinned to 2add963021)

Solutions

1. Validate every source path exists (hdfs dfs -ls or getFileInfo batch) before calling concat
2. Rebuild the source list from a fresh listing of the target directory at concat time
3. Fix mount entries if sources exist in a subcluster but are not visible through the Router

// Validate all sources exist before concat:
for (String s : srcs) {
  if (dfs.getClient().getNamenode().getFileInfo(s) == null) {
    throw new IllegalArgumentException("missing source: " + s);
  }
}

catch (IOException e) {
  if (e.getMessage().startsWith("Cannot find source file - ")) {
    // identify the missing part file, regenerate/skip it, rebuild src list
  }
}

When it happens

Trigger: concat(trg, srcs) where any entry of srcs does not exist (typo, deleted, or still being written); a source path resolving to a namespace where the file is absent.

Common situations: Job commit concatenating part files when one part was cleaned up or failed to flush; partial uploads; source list built from a stale directory listing.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• Cannot find target file - {trg}
• Cannot concatenate source file {sourceFile} because it is lo
• %s does not exist or is not file.
• File {} not found.
• No namespace availaible.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c94db295e16222e8. Report an issue: GitHub.