{"record":{"id":"cfdef9e77da546bf","repo":"apache/hadoop","slug":"concat-source-file-src-is-invalid-or-emp","errorCode":null,"errorMessage":"\"concat: source file \" + src + \" is invalid or empty or underConstruction\"","messagePattern":"\"concat: source file \" \\+ src \\+ \" is invalid or empty or underConstruction\"","errorType":"exception","errorClass":"HadoopIllegalArgumentException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java","lineNumber":154,"sourceCode":"      // make sure all the source files are not in snapshot\n      if (srcINode.isInLatestSnapshot(iip.getLatestSnapshotId())) {\n        throw new SnapshotException(\"Concat: the source file \" + src\n            + \" is in snapshot\");\n      }\n      // check if the file has other references.\n      if (srcINode.isReference() && ((INodeReference.WithCount)\n          srcINode.asReference().getReferredINode()).getReferenceCount() > 1) {\n        throw new SnapshotException(\"Concat: the source file \" + src\n            + \" is referred by some other reference in some snapshot.\");\n      }\n      // source file cannot be the same with the target file\n      if (srcINode.equals(targetINode)) {\n        throw new HadoopIllegalArgumentException(\"concat: the src file \" + src\n            + \" is the same with the target file \" + targetIIP.getPath());\n      }\n      // source file cannot be under construction or empty\n      if(srcINodeFile.isUnderConstruction() || srcINodeFile.numBlocks() == 0) {\n        throw new HadoopIllegalArgumentException(\"concat: source file \" + src\n            + \" is invalid or empty or underConstruction\");\n      }\n\n      // source file's preferred block size cannot be greater than the target\n      // file\n      if (srcINodeFile.getPreferredBlockSize() >\n          targetINode.getPreferredBlockSize()) {\n        throw new HadoopIllegalArgumentException(\"concat: source file \" + src\n            + \" has preferred block size \" + srcINodeFile.getPreferredBlockSize()\n            + \" which is greater than the target file's preferred block size \"\n            + targetINode.getPreferredBlockSize());\n      }\n      if(srcINodeFile.getErasureCodingPolicyID() !=\n          targetINode.getErasureCodingPolicyID()) {\n        throw new HadoopIllegalArgumentException(\"Source file \" + src\n            + \" and target file \" + targetIIP.getPath()\n            + \" have different erasure coding policy\");\n      }","sourceCodeStart":136,"sourceCodeEnd":172,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java#L136-L172","documentation":"Each source must be a finalized, non-empty regular file: verifySrcFiles rejects srcs where isUnderConstruction() or numBlocks() == 0 with HadoopIllegalArgumentException. Concat splices existing block lists — an open file's block list is still mutating, and an empty file contributes nothing while still requiring inode deletion, so both are rejected up front.","triggerScenarios":"Passing a still-open file (writer has not closed / lease not recovered) as a src; passing a zero-length file (created but never written, or truncated); passing a path that is a directory (INodeFile.valueOf would fail adjacent to this check).","commonSituations":"Rolling-writer jobs where the current output file is accidentally included in the concat list; empty part files produced by failed map/reduce tasks; compaction globs that match marker/zero-byte files.","solutions":["Close (or recoverLease for crashed writers) every src before concat.","Filter srcs by st.getLen() > 0 using getFileStatus — zero-length files have zero blocks.","Filter out directories and anything that is not a regular file before building the src array."],"exampleFix":"// before\nfs.concat(target, dirListing(dir)); // may contain empty or open files\n\n// after\nList<Path> srcs = new ArrayList<>();\nfor (FileStatus st : fs.listStatus(dir)) {\n  if (st.isFile() && st.getLen() > 0 && !st.getPath().equals(target)) {\n    srcs.add(st.getPath());\n  } // skip empty files and directories\n}\nif (!srcs.isEmpty()) fs.concat(target, srcs.toArray(new Path[0]));","handlingStrategy":"validation","validationCode":"List<Path> ok = new ArrayList<>();\nfor (Path src : srcs) {\n  FileStatus st = fs.getFileStatus(src);\n  if (st.isFile() && st.getLen() > 0) ok.add(src); // 0 length => 0 blocks\n}","typeGuard":null,"tryCatchPattern":"catch (HadoopIllegalArgumentException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"invalid or empty or underConstruction\")) {\n    recoverLeasesAndFilterEmpty(srcs); // close/recover open files, drop empties, retry concat\n  } else { throw e; }\n}","preventionTips":["Filter zero-length files out of concat candidates before the call (len == 0 implies numBlocks == 0).","Ensure all src writers are closed; call recoverLease for any file from a crashed process.","Have producers delete their own empty marker files instead of leaving them for compaction to trip over."],"tags":["hdfs","concat","empty-file","under-construction","precondition"],"backgroundTag":"concat-invalid-source","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}