{"record":{"id":"a275bdd905b27417","repo":"apache/hadoop","slug":"append-on-ec-file-without-new-block-is-not-support","errorCode":null,"errorMessage":"Append on EC file without new block is not supported. Use NEW_BLOCK create flag while appending file.","messagePattern":"Append on EC file without new block is not supported\\. Use NEW_BLOCK create flag while appending file\\.","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java","lineNumber":113,"sourceCode":"      final INode inode = iip.getLastINode();\n      final String path = iip.getPath();\n      if (inode != null && inode.isDirectory()) {\n        throw new FileAlreadyExistsException(\"Cannot append to directory \"\n            + path + \"; already exists as a directory.\");\n      }\n      if (fsd.isPermissionEnabled()) {\n        fsd.checkPathAccess(pc, iip, FsAction.WRITE);\n      }\n\n      if (inode == null) {\n        throw new FileNotFoundException(\n            \"Failed to append to non-existent file \" + path + \" for client \"\n                + clientMachine);\n      }\n      final INodeFile file = INodeFile.valueOf(inode, path, true);\n\n      if (file.isStriped() && !newBlock) {\n        throw new UnsupportedOperationException(\n            \"Append on EC file without new block is not supported. Use \"\n                + CreateFlag.NEW_BLOCK + \" create flag while appending file.\");\n      }\n\n      BlockManager blockManager = fsd.getBlockManager();\n      final BlockStoragePolicy lpPolicy = blockManager\n          .getStoragePolicy(\"LAZY_PERSIST\");\n      if (lpPolicy != null && lpPolicy.getId() == file.getStoragePolicyID()) {\n        throw new UnsupportedOperationException(\n            \"Cannot append to lazy persist file \" + path);\n      }\n      // Opening an existing file for append - may need to recover lease.\n      fsn.recoverLeaseInternal(RecoverLeaseOp.APPEND_FILE, iip, path, holder,\n          clientMachine, false);\n\n      final BlockInfo lastBlock = file.getLastBlock();\n      // Check that the block has at least minimum replication.\n      if (lastBlock != null) {","sourceCodeStart":95,"sourceCodeEnd":131,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java#L95-L131","documentation":"Erasure-coded (striped) files cannot be extended in place; the NameNode only supports append on an EC file by adding a new striped block. FSDirAppendOp throws UnsupportedOperationException when file.isStriped() and the client did not pass CreateFlag.NEW_BLOCK - the message itself tells you the required flag.","triggerScenarios":"DistributedFileSystem.append(path) without EnumSet.of(CreateFlag.APPEND, CreateFlag.NEW_BLOCK) on a file laid out with an EC policy (directory covered by 'hdfs ec -enablePolicy' / '-setErasureCodingPolicy', or file created with an EC policy).","commonSituations":"Enabling EC on directory trees that existing generic append-based writers then target; Hadoop 2 to 3 migrations where EC zones appear around append-heavy workloads; frameworks (Spark/Hive appenders, Flume HDFS sink) that do not send NEW_BLOCK.","solutions":["Pass NEW_BLOCK: ((DistributedFileSystem) fs).append(path, bufferSize, progress, EnumSet.of(CreateFlag.APPEND, CreateFlag.NEW_BLOCK)).","If the writer framework cannot be changed, keep the target file on replicated layout: create it outside EC dirs or set the parent dir back to replicated ('hdfs ec -setErasureCodingPolicy -replicated <dir>').","For framework authors: detect EC layout and choose flags automatically so append works on both file types."],"exampleFix":"// before\nFSDataOutputStream out = fs.append(path); // fails on EC files\n\n// after\nEnumSet<CreateFlag> flags = isErasureCoded(path)\n    ? EnumSet.of(CreateFlag.APPEND, CreateFlag.NEW_BLOCK)\n    : EnumSet.of(CreateFlag.APPEND);\nFSDataOutputStream out = ((DistributedFileSystem) fs).append(path, 128 * 1024, null, flags);","handlingStrategy":"validation","validationCode":"HdfsAdmin admin = new HdfsAdmin(path.toUri(), fs.getConf());\nErasureCodingPolicy ecp = admin.getErasureCodingPolicy(path.getParent());\nEnumSet<CreateFlag> flags = (ecp == null)\n    ? EnumSet.of(CreateFlag.APPEND)\n    : EnumSet.of(CreateFlag.APPEND, CreateFlag.NEW_BLOCK);\nFSDataOutputStream out = ((DistributedFileSystem) fs).append(path, 128 * 1024, null, flags);","typeGuard":null,"tryCatchPattern":"try {\n  out = dfs.append(path, bufSize, null, EnumSet.of(CreateFlag.APPEND, CreateFlag.NEW_BLOCK));\n} catch (UnsupportedOperationException e) {\n  if (e.getMessage().contains(\"NEW_BLOCK\")) {\n    // caller did not send NEW_BLOCK on an EC file: fix flags, never retry unchanged\n  }\n  throw e;\n}","preventionTips":["Keep files targeted by generic appenders outside EC zones.","Centralize append logic in one utility that queries the EC policy and picks flags."],"tags":["hdfs","append","erasure-coding","create-flag"],"backgroundTag":"erasure-coding-limitation","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}