{"record":{"id":"65115fe3a5097ee0","repo":"apache/hadoop","slug":"source-file-src-and-target-file-targe","errorCode":null,"errorMessage":"\"Source file \" + src + \" and target file \" + targetIIP.getPath() + \" have different erasure coding policy\"","messagePattern":"\"Source file \" \\+ src \\+ \" and target file \" \\+ targetIIP\\.getPath\\(\\) \\+ \" have different erasure coding policy\"","errorType":"exception","errorClass":"HadoopIllegalArgumentException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java","lineNumber":169,"sourceCode":"      }\n      // source file cannot be under construction or empty\n      if(srcINodeFile.isUnderConstruction() || srcINodeFile.numBlocks() == 0) {\n        throw new HadoopIllegalArgumentException(\"concat: source file \" + src\n            + \" is invalid or empty or underConstruction\");\n      }\n\n      // source file's preferred block size cannot be greater than the target\n      // file\n      if (srcINodeFile.getPreferredBlockSize() >\n          targetINode.getPreferredBlockSize()) {\n        throw new HadoopIllegalArgumentException(\"concat: source file \" + src\n            + \" has preferred block size \" + srcINodeFile.getPreferredBlockSize()\n            + \" which is greater than the target file's preferred block size \"\n            + targetINode.getPreferredBlockSize());\n      }\n      if(srcINodeFile.getErasureCodingPolicyID() !=\n          targetINode.getErasureCodingPolicyID()) {\n        throw new HadoopIllegalArgumentException(\"Source file \" + src\n            + \" and target file \" + targetIIP.getPath()\n            + \" have different erasure coding policy\");\n      }\n      si.add(srcINodeFile);\n    }\n\n    // make sure no two files are the same\n    if(si.size() < srcs.length) {\n      // it means at least two files are the same\n      throw new HadoopIllegalArgumentException(\n          \"concat: at least two of the source files are the same\");\n    }\n    return si.toArray(new INodeFile[si.size()]);\n  }\n\n  private static QuotaCounts computeQuotaDeltas(FSDirectory fsd,\n      INodeFile target, INodeFile[] srcList) {\n    QuotaCounts deltas = new QuotaCounts.Builder().build();","sourceCodeStart":151,"sourceCodeEnd":187,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java#L151-L187","documentation":"Concat requires src and target to have the same erasure coding policy (both replicated, or the same EC schema): verifySrcFiles compares getErasureCodingPolicyID() and throws on mismatch. Blocks of an EC-coded file hold parity-stripe cells, not plain replicas, and cannot be appended to a file with a different — or replicated — layout without rewriting the data.","triggerScenarios":"Concat where the target is replicated and a src is on an EC policy directory (or srcs/target use different EC schemas, e.g. RS-3-2-1024k vs RS-6-3-1024k).","commonSituations":"Enabling erasure coding on a directory while a compaction job merges it with replicated directories; mixing files written before and after an EC policy was applied to the directory; cross-directory compaction where one directory is EC and the other replicated.","solutions":["Group files by EC policy (query with DistributedFileSystem.getErasureCodingPolicy) and concat only within the same policy.","Recreate mismatched files under the target's policy (set via fs.setErasureCodingPolicy on the directory) and rewrite the data before concat.","Make the compaction job policy-aware: for cross-policy merges fall back to copy+delete."],"exampleFix":"// before\nfs.concat(replicatedTarget, allSources); // one src is EC-coded -> error\n\n// after\nErasureCodingPolicy tgtEc = dfs.getErasureCodingPolicy(target);\nPath[] samePolicy = Arrays.stream(srcs)\n    .filter(p -> Objects.equals(dfs.getErasureCodingPolicy(p), tgtEc))\n    .toArray(Path[]::new);\nfs.concat(target, samePolicy); // rewrite cross-policy files instead","handlingStrategy":"validation","validationCode":"DistributedFileSystem dfs = (DistributedFileSystem) fs;\nErasureCodingPolicy tgtEc = dfs.getErasureCodingPolicy(target);\nList<Path> samePolicy = new ArrayList<>();\nfor (Path src : srcs) {\n  if (Objects.equals(dfs.getErasureCodingPolicy(src), tgtEc)) samePolicy.add(src);\n}","typeGuard":null,"tryCatchPattern":"catch (HadoopIllegalArgumentException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"erasure coding policy\")) {\n    // partition srcs by EC policy and concat only matching groups\n    concatByPolicy(target, srcs);\n  } else { throw e; }\n}","preventionTips":["Query the EC policy of target and every src during planning (getErasureCodingPolicy) and group accordingly.","Never mix replicated and EC-coded directories in one compaction batch.","When enabling EC on a directory, update downstream concat pipelines in the same change."],"tags":["hdfs","concat","erasure-coding","policy-mismatch","precondition"],"backgroundTag":"erasure-coding-policy-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}