apache/hadoop · error · HadoopIllegalArgumentException

concat can not be called for files in an encryption zone.

Error message

concat can not be called for files in an encryption zone.

What it means

Concat is permanently unsupported for files inside an encryption zone: FSDirConcatOp.verifyTargetFile looks up the target's encryption zone via FSDirEncryptionZoneOp.getEZForPath and throws HadoopIllegalArgumentException if one exists. Each encrypted file has its own per-file EDEK, so appending one file's blocks to another would produce ciphertext no key can decrypt — the NameNode therefore rejects the operation outright rather than corrupt data.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java:103

    if (FSDirectory.isReservedRawName(target)
        || FSDirectory.isReservedInodesName(target)) {
      throw new IOException("Concat operation doesn't support "
          + FSDirectory.DOT_RESERVED_STRING + " relative path : " + target);
    }
    for (String srcPath : srcs) {
      if (FSDirectory.isReservedRawName(srcPath)
          || FSDirectory.isReservedInodesName(srcPath)) {
        throw new IOException("Concat operation doesn't support "
            + FSDirectory.DOT_RESERVED_STRING + " relative path : " + srcPath);
      }
    }
  }

  private static void verifyTargetFile(FSDirectory fsd, final String target,
      final INodesInPath targetIIP) throws IOException {
    // check the target
    if (FSDirEncryptionZoneOp.getEZForPath(fsd, targetIIP) != null) {
      throw new HadoopIllegalArgumentException(
          "concat can not be called for files in an encryption zone.");
    }
    final INodeFile targetINode = INodeFile.valueOf(targetIIP.getLastINode(),
        target);
    if(targetINode.isUnderConstruction()) {
      throw new HadoopIllegalArgumentException("concat: target file "
          + target + " is under construction");
    }
  }

  private static INodeFile[] verifySrcFiles(FSDirectory fsd, String[] srcs,
      INodesInPath targetIIP, FSPermissionChecker pc) throws IOException {
    // to make sure no two files are the same
    Set<INodeFile> si = new LinkedHashSet<>();
    final INodeFile targetINode = targetIIP.getLastINode().asFile();
    final INodeDirectory targetParent = targetINode.getParent();
    // now check the srcs
    for(String src : srcs) {

View on GitHub (pinned to 2add963021)

Solutions

Exclude encryption-zone paths from concat-based compaction — copy-and-delete (stream through the client) or HDFS Federation/appending at write time are the alternatives inside a zone.
Detect zones up front with HdfsAdmin.getEncryptionZoneForPath and route such files to a different strategy in your pipeline.
If the data does not need encryption, recreate it outside the zone and concat there.

Example fix

// before
fs.concat(target, srcs); // target inside an EZ -> HadoopIllegalArgumentException

// after
HdfsAdmin admin = new HdfsAdmin(fs.getUri(), conf);
if (admin.getEncryptionZoneForPath(target) != null) {
  compactByCopy(target, srcs); // stream srcs into target, then delete srcs
} else {
  fs.concat(target, srcs);
}

Defensive patterns

Strategy: validation

Validate before calling

HdfsAdmin admin = new HdfsAdmin(fs.getUri(), conf);
if (admin.getEncryptionZoneForPath(target) != null) {
  // concat unsupported here; use a copy-merge strategy instead
  compactByCopy(target, srcs);
  return;
}

Try / catch

catch (HadoopIllegalArgumentException e) {
  if (e.getMessage() != null && e.getMessage().contains("encryption zone")) {
    // route to a copy-based compaction; do NOT retry concat
    compactByCopy(target, srcs);
  } else { throw e; }
}

Prevention

Query zone membership (HdfsAdmin.getEncryptionZoneForPath) during planning, before the compaction job runs.
Keep concat-based compaction scoped to directories known to be outside all encryption zones.
When enabling TDE on an existing dataset, audit compaction pipelines for concat usage first.

When it happens

Trigger: FileSystem.concat(target, srcs) where the target resolves inside any encryption zone (the zone root or any descendant directory). The check fires before under-construction or source checks.

Common situations: Generic compaction jobs (e.g. HBase-style small-file merging, home-grown Spark/Hadoop combiners) run across the whole warehouse and hit an EZ created for a compliance directory; teams enable HDFS TDE partway through and existing concat pipelines start failing only for that subtree.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d58113adb2cd8a54. Report an issue: GitHub.