{"record":{"id":"c94db295e16222e8","repo":"apache/hadoop","slug":"cannot-find-source-file-sourcefile","errorCode":null,"errorMessage":"Cannot find source file - {sourceFile}","messagePattern":"Cannot find source file - (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/async/RouterAsyncClientProtocol.java","lineNumber":329,"sourceCode":"\n  @Override\n  public void concat(String trg, String[] src) throws IOException {\n    rpcServer.checkOperation(NameNode.OperationCategory.WRITE);\n\n    // Concat only effects when all files in the same namespace.\n    getFileRemoteLocation(trg);\n    asyncApply((AsyncApplyFunction<RemoteLocation, Object>) targetDestination -> {\n      if (targetDestination == null) {\n        throw new IOException(\"Cannot find target file - \" + trg);\n      }\n      String targetNameService = targetDestination.getNameserviceId();\n      String[] sourceDestinations = new String[src.length];\n      int[] index = new int[1];\n      asyncForEach(Arrays.stream(src).iterator(), (forEachRun, sourceFile) -> {\n        getFileRemoteLocation(sourceFile);\n        asyncApply((ApplyFunction<RemoteLocation, Object>) srcLocation -> {\n          if (srcLocation == null) {\n            throw new IOException(\"Cannot find source file - \" + sourceFile);\n          }\n          sourceDestinations[index[0]++] = srcLocation.getDest();\n          if (!targetNameService.equals(srcLocation.getNameserviceId())) {\n            throw new IOException(\"Cannot concatenate source file \" + sourceFile\n                + \" because it is located in a different namespace\" + \" with nameservice \"\n                + srcLocation.getNameserviceId() + \" from the target file with nameservice \"\n                + targetNameService);\n          }\n          return null;\n        });\n      });\n      asyncApply((AsyncApplyFunction<Object, Object>) o -> {\n        // Invoke\n        RemoteMethod method = new RemoteMethod(\"concat\",\n            new Class<?>[] {String.class, String[].class},\n            targetDestination.getDest(), sourceDestinations);\n        rpcClient.invokeSingle(targetDestination, method, Void.class);\n      });","sourceCodeStart":311,"sourceCodeEnd":347,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/async/RouterAsyncClientProtocol.java#L311-L347","documentation":"Companion check in RouterAsyncClientProtocol.concat: while iterating the source files, each srcLocation is resolved with getFileRemoteLocation; null means that source file could not be found in any namespace and IOException('Cannot find source file - <sourceFile>') aborts the operation. One missing source blocks the whole concat because the operation must atomically merge all listed files into the target.","triggerScenarios":"concat(trg, srcs) where any entry of srcs does not exist (typo, deleted, or still being written); a source path resolving to a namespace where the file is absent.","commonSituations":"Job commit concatenating part files when one part was cleaned up or failed to flush; partial uploads; source list built from a stale directory listing.","solutions":["Validate every source path exists (hdfs dfs -ls or getFileInfo batch) before calling concat","Rebuild the source list from a fresh listing of the target directory at concat time","Fix mount entries if sources exist in a subcluster but are not visible through the Router"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Validate all sources exist before concat:\nfor (String s : srcs) {\n  if (dfs.getClient().getNamenode().getFileInfo(s) == null) {\n    throw new IllegalArgumentException(\"missing source: \" + s);\n  }\n}","typeGuard":null,"tryCatchPattern":"catch (IOException e) {\n  if (e.getMessage().startsWith(\"Cannot find source file - \")) {\n    // identify the missing part file, regenerate/skip it, rebuild src list\n  }\n}","preventionTips":["Re-list the part-file directory at concat time instead of trusting an earlier listing","Ensure all writers finished (lease recovery) before building the concat source array","Fail fast with explicit missing-file diagnostics in job frameworks that concat outputs"],"tags":["hadoop","hdfs","rbf","concat","file-not-found","async-rpc"],"backgroundTag":"file-not-found","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}