{"record":{"id":"d58113adb2cd8a54","repo":"apache/hadoop","slug":"concat-can-not-be-called-for-files-in-an-encryptio","errorCode":null,"errorMessage":"concat can not be called for files in an encryption zone.","messagePattern":"concat can not be called for files in an encryption zone\\.","errorType":"exception","errorClass":"HadoopIllegalArgumentException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java","lineNumber":103,"sourceCode":"    if (FSDirectory.isReservedRawName(target)\n        || FSDirectory.isReservedInodesName(target)) {\n      throw new IOException(\"Concat operation doesn't support \"\n          + FSDirectory.DOT_RESERVED_STRING + \" relative path : \" + target);\n    }\n    for (String srcPath : srcs) {\n      if (FSDirectory.isReservedRawName(srcPath)\n          || FSDirectory.isReservedInodesName(srcPath)) {\n        throw new IOException(\"Concat operation doesn't support \"\n            + FSDirectory.DOT_RESERVED_STRING + \" relative path : \" + srcPath);\n      }\n    }\n  }\n\n  private static void verifyTargetFile(FSDirectory fsd, final String target,\n      final INodesInPath targetIIP) throws IOException {\n    // check the target\n    if (FSDirEncryptionZoneOp.getEZForPath(fsd, targetIIP) != null) {\n      throw new HadoopIllegalArgumentException(\n          \"concat can not be called for files in an encryption zone.\");\n    }\n    final INodeFile targetINode = INodeFile.valueOf(targetIIP.getLastINode(),\n        target);\n    if(targetINode.isUnderConstruction()) {\n      throw new HadoopIllegalArgumentException(\"concat: target file \"\n          + target + \" is under construction\");\n    }\n  }\n\n  private static INodeFile[] verifySrcFiles(FSDirectory fsd, String[] srcs,\n      INodesInPath targetIIP, FSPermissionChecker pc) throws IOException {\n    // to make sure no two files are the same\n    Set<INodeFile> si = new LinkedHashSet<>();\n    final INodeFile targetINode = targetIIP.getLastINode().asFile();\n    final INodeDirectory targetParent = targetINode.getParent();\n    // now check the srcs\n    for(String src : srcs) {","sourceCodeStart":85,"sourceCodeEnd":121,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java#L85-L121","documentation":"Concat is permanently unsupported for files inside an encryption zone: FSDirConcatOp.verifyTargetFile looks up the target's encryption zone via FSDirEncryptionZoneOp.getEZForPath and throws HadoopIllegalArgumentException if one exists. Each encrypted file has its own per-file EDEK, so appending one file's blocks to another would produce ciphertext no key can decrypt — the NameNode therefore rejects the operation outright rather than corrupt data.","triggerScenarios":"FileSystem.concat(target, srcs) where the target resolves inside any encryption zone (the zone root or any descendant directory). The check fires before under-construction or source checks.","commonSituations":"Generic compaction jobs (e.g. HBase-style small-file merging, home-grown Spark/Hadoop combiners) run across the whole warehouse and hit an EZ created for a compliance directory; teams enable HDFS TDE partway through and existing concat pipelines start failing only for that subtree.","solutions":["Exclude encryption-zone paths from concat-based compaction — copy-and-delete (stream through the client) or HDFS Federation/appending at write time are the alternatives inside a zone.","Detect zones up front with HdfsAdmin.getEncryptionZoneForPath and route such files to a different strategy in your pipeline.","If the data does not need encryption, recreate it outside the zone and concat there."],"exampleFix":"// before\nfs.concat(target, srcs); // target inside an EZ -> HadoopIllegalArgumentException\n\n// after\nHdfsAdmin admin = new HdfsAdmin(fs.getUri(), conf);\nif (admin.getEncryptionZoneForPath(target) != null) {\n  compactByCopy(target, srcs); // stream srcs into target, then delete srcs\n} else {\n  fs.concat(target, srcs);\n}","handlingStrategy":"validation","validationCode":"HdfsAdmin admin = new HdfsAdmin(fs.getUri(), conf);\nif (admin.getEncryptionZoneForPath(target) != null) {\n  // concat unsupported here; use a copy-merge strategy instead\n  compactByCopy(target, srcs);\n  return;\n}","typeGuard":null,"tryCatchPattern":"catch (HadoopIllegalArgumentException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"encryption zone\")) {\n    // route to a copy-based compaction; do NOT retry concat\n    compactByCopy(target, srcs);\n  } else { throw e; }\n}","preventionTips":["Query zone membership (HdfsAdmin.getEncryptionZoneForPath) during planning, before the compaction job runs.","Keep concat-based compaction scoped to directories known to be outside all encryption zones.","When enabling TDE on an existing dataset, audit compaction pipelines for concat usage first."],"tags":["hdfs","concat","encryption-zone","unsupported-operation"],"backgroundTag":"concat-in-encryption-zone","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}