apache/hadoop · error · HadoopIllegalArgumentException

"Source file " + src + " and target file " + targetIIP.getPa

Error message

"Source file " + src + " and target file " + targetIIP.getPath() + " have different erasure coding policy"

What it means

Concat requires src and target to have the same erasure coding policy (both replicated, or the same EC schema): verifySrcFiles compares getErasureCodingPolicyID() and throws on mismatch. Blocks of an EC-coded file hold parity-stripe cells, not plain replicas, and cannot be appended to a file with a different — or replicated — layout without rewriting the data.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java:169

      }
      // source file cannot be under construction or empty
      if(srcINodeFile.isUnderConstruction() || srcINodeFile.numBlocks() == 0) {
        throw new HadoopIllegalArgumentException("concat: source file " + src
            + " is invalid or empty or underConstruction");
      }

      // source file's preferred block size cannot be greater than the target
      // file
      if (srcINodeFile.getPreferredBlockSize() >
          targetINode.getPreferredBlockSize()) {
        throw new HadoopIllegalArgumentException("concat: source file " + src
            + " has preferred block size " + srcINodeFile.getPreferredBlockSize()
            + " which is greater than the target file's preferred block size "
            + targetINode.getPreferredBlockSize());
      }
      if(srcINodeFile.getErasureCodingPolicyID() !=
          targetINode.getErasureCodingPolicyID()) {
        throw new HadoopIllegalArgumentException("Source file " + src
            + " and target file " + targetIIP.getPath()
            + " have different erasure coding policy");
      }
      si.add(srcINodeFile);
    }

    // make sure no two files are the same
    if(si.size() < srcs.length) {
      // it means at least two files are the same
      throw new HadoopIllegalArgumentException(
          "concat: at least two of the source files are the same");
    }
    return si.toArray(new INodeFile[si.size()]);
  }

  private static QuotaCounts computeQuotaDeltas(FSDirectory fsd,
      INodeFile target, INodeFile[] srcList) {
    QuotaCounts deltas = new QuotaCounts.Builder().build();

View on GitHub (pinned to 2add963021)

Solutions

Group files by EC policy (query with DistributedFileSystem.getErasureCodingPolicy) and concat only within the same policy.
Recreate mismatched files under the target's policy (set via fs.setErasureCodingPolicy on the directory) and rewrite the data before concat.
Make the compaction job policy-aware: for cross-policy merges fall back to copy+delete.

Example fix

// before
fs.concat(replicatedTarget, allSources); // one src is EC-coded -> error

// after
ErasureCodingPolicy tgtEc = dfs.getErasureCodingPolicy(target);
Path[] samePolicy = Arrays.stream(srcs)
    .filter(p -> Objects.equals(dfs.getErasureCodingPolicy(p), tgtEc))
    .toArray(Path[]::new);
fs.concat(target, samePolicy); // rewrite cross-policy files instead

Defensive patterns

Strategy: validation

Validate before calling

DistributedFileSystem dfs = (DistributedFileSystem) fs;
ErasureCodingPolicy tgtEc = dfs.getErasureCodingPolicy(target);
List<Path> samePolicy = new ArrayList<>();
for (Path src : srcs) {
  if (Objects.equals(dfs.getErasureCodingPolicy(src), tgtEc)) samePolicy.add(src);
}

Try / catch

catch (HadoopIllegalArgumentException e) {
  if (e.getMessage() != null && e.getMessage().contains("erasure coding policy")) {
    // partition srcs by EC policy and concat only matching groups
    concatByPolicy(target, srcs);
  } else { throw e; }
}

Prevention

Query the EC policy of target and every src during planning (getErasureCodingPolicy) and group accordingly.
Never mix replicated and EC-coded directories in one compaction batch.
When enabling EC on a directory, update downstream concat pipelines in the same change.

When it happens

Trigger: Concat where the target is replicated and a src is on an EC policy directory (or srcs/target use different EC schemas, e.g. RS-3-2-1024k vs RS-6-3-1024k).

Common situations: Enabling erasure coding on a directory while a compaction job merges it with replicated directories; mixing files written before and after an EC policy was applied to the directory; cross-directory compaction where one directory is EC and the other replicated.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/65115fe3a5097ee0. Report an issue: GitHub.