{"record":{"id":"aea162e234832785","repo":"apache/hadoop","slug":"concatdeleteop-can-only-have-sources-at-most","errorCode":null,"errorMessage":"ConcatDeleteOp can only have {} sources at most.","messagePattern":"ConcatDeleteOp can only have (.+?) sources at most\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java","lineNumber":1275,"sourceCode":"      return cache.get(OP_CONCAT_DELETE);\n    }\n\n    @Override\n    void resetSubFields() {\n      length = 0;\n      trg = null;\n      srcs = null;\n      timestamp = 0L;\n    }\n\n    ConcatDeleteOp setTarget(String trg) {\n      this.trg = trg;\n      return this;\n    }\n\n    ConcatDeleteOp setSources(String[] srcs) {\n      if (srcs.length > MAX_CONCAT_SRC) {\n        throw new RuntimeException(\"ConcatDeleteOp can only have \" +\n            MAX_CONCAT_SRC + \" sources at most.\");\n      }\n      this.srcs = srcs;\n\n      return this;\n    }\n\n    ConcatDeleteOp setTimestamp(long timestamp) {\n      this.timestamp = timestamp;\n      return this;\n    }\n\n    @Override\n    public void writeFields(DataOutputStream out) throws IOException {\n      FSImageSerialization.writeString(trg, out);\n            \n      DeprecatedUTF8 info[] = new DeprecatedUTF8[srcs.length];\n      int idx = 0;","sourceCodeStart":1257,"sourceCodeEnd":1293,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java#L1257-L1293","documentation":"ConcatDeleteOp.setSources() guards the source list of a concat edit record: MAX_CONCAT_SRC is 1024*1024 = 1,048,576 (FSEditLogOp.java:1250), and passing more sources throws this RuntimeException before the record is ever written to the edit log. It fires on the NameNode while building the edit op for a concat RPC, so the operation fails atomically rather than logging an unreadable record.","triggerScenarios":"A client calls DistributedFileSystem.concat(target, srcs) / ClientProtocol.concat with srcs.length > 1,048,576; the NameNode constructs the ConcatDeleteOp via setSources() and throws. In practice only generated code or a loop that expands a huge directory listing into one call produces this.","commonSituations":"Migration/compaction scripts that try to concatenate an entire directory tree in a single concat call; glob expansion feeding an unbounded array into concat.","solutions":["Split the work into batches of at most 1,048,576 sources: concat the first batch into the target, then concat each further batch into the same target","Validate srcs.length in the client before issuing the RPC and fail fast with a clear message","Check for a caller bug (accidental nesting/duplication) that inflated the source array"],"exampleFix":"// before\nPath[] all = listAllFiles(srcDir);\nfs.concat(trg, all);  // RuntimeException: ConcatDeleteOp can only have 1048576 sources at most.\n\n// after\nstatic final int MAX_CONCAT_SRC = 1024 * 1024;\nfor (int i = 0; i < all.length; i += MAX_CONCAT_SRC) {\n  Path[] batch = Arrays.copyOfRange(all, i, Math.min(i + MAX_CONCAT_SRC, all.length));\n  fs.concat(trg, batch);  // first batch merges into trg; later batches merge into the grown trg\n}","handlingStrategy":"validation","validationCode":"static final int MAX_CONCAT_SRC = 1024 * 1024; // must match FSEditLogOp.MAX_CONCAT_SRC\nif (srcs == null || srcs.length == 0 || srcs.length > MAX_CONCAT_SRC) {\n  throw new IllegalArgumentException(\n      \"concat sources must be in [1, \" + MAX_CONCAT_SRC + \"], got \" + (srcs == null ? 0 : srcs.length));\n}\nfs.concat(trg, srcs);","typeGuard":"boolean isConcatSourceCountValid(int n) {\n  return n >= 1 && n <= 1024 * 1024; // HDFS hard cap, FSEditLogOp.java:1250\n}","tryCatchPattern":"try {\n  fs.concat(trg, srcs);\n} catch (RuntimeException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"sources at most\")) {\n    // split into batches of <= 1048576 and retry\n  } else { throw e; }\n}","preventionTips":["Never pass an unbounded glob/listing result into concat; cap it in client code","Batch large merges at 1,048,576 sources or fewer per call","Fail fast with a clear client-side message instead of relying on the NameNode RuntimeException"],"tags":["hdfs","concat","edit-log","validation","limits"],"backgroundTag":"argument-limit-exceeded","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}