apache/hadoop · error · UnsupportedOperationException

Append on EC file without new block is not supported. Use NE

Error message

Append on EC file without new block is not supported. Use NEW_BLOCK create flag while appending file.

What it means

Erasure-coded (striped) files cannot be extended in place; the NameNode only supports append on an EC file by adding a new striped block. FSDirAppendOp throws UnsupportedOperationException when file.isStriped() and the client did not pass CreateFlag.NEW_BLOCK - the message itself tells you the required flag.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java:113

      final INode inode = iip.getLastINode();
      final String path = iip.getPath();
      if (inode != null && inode.isDirectory()) {
        throw new FileAlreadyExistsException("Cannot append to directory "
            + path + "; already exists as a directory.");
      }
      if (fsd.isPermissionEnabled()) {
        fsd.checkPathAccess(pc, iip, FsAction.WRITE);
      }

      if (inode == null) {
        throw new FileNotFoundException(
            "Failed to append to non-existent file " + path + " for client "
                + clientMachine);
      }
      final INodeFile file = INodeFile.valueOf(inode, path, true);

      if (file.isStriped() && !newBlock) {
        throw new UnsupportedOperationException(
            "Append on EC file without new block is not supported. Use "
                + CreateFlag.NEW_BLOCK + " create flag while appending file.");
      }

      BlockManager blockManager = fsd.getBlockManager();
      final BlockStoragePolicy lpPolicy = blockManager
          .getStoragePolicy("LAZY_PERSIST");
      if (lpPolicy != null && lpPolicy.getId() == file.getStoragePolicyID()) {
        throw new UnsupportedOperationException(
            "Cannot append to lazy persist file " + path);
      }
      // Opening an existing file for append - may need to recover lease.
      fsn.recoverLeaseInternal(RecoverLeaseOp.APPEND_FILE, iip, path, holder,
          clientMachine, false);

      final BlockInfo lastBlock = file.getLastBlock();
      // Check that the block has at least minimum replication.
      if (lastBlock != null) {

View on GitHub (pinned to 2add963021)

Solutions

1. Pass NEW_BLOCK: ((DistributedFileSystem) fs).append(path, bufferSize, progress, EnumSet.of(CreateFlag.APPEND, CreateFlag.NEW_BLOCK)).
2. If the writer framework cannot be changed, keep the target file on replicated layout: create it outside EC dirs or set the parent dir back to replicated ('hdfs ec -setErasureCodingPolicy -replicated <dir>').
3. For framework authors: detect EC layout and choose flags automatically so append works on both file types.

Example fix

// before
FSDataOutputStream out = fs.append(path); // fails on EC files

// after
EnumSet<CreateFlag> flags = isErasureCoded(path)
    ? EnumSet.of(CreateFlag.APPEND, CreateFlag.NEW_BLOCK)
    : EnumSet.of(CreateFlag.APPEND);
FSDataOutputStream out = ((DistributedFileSystem) fs).append(path, 128 * 1024, null, flags);

HdfsAdmin admin = new HdfsAdmin(path.toUri(), fs.getConf());
ErasureCodingPolicy ecp = admin.getErasureCodingPolicy(path.getParent());
EnumSet<CreateFlag> flags = (ecp == null)
    ? EnumSet.of(CreateFlag.APPEND)
    : EnumSet.of(CreateFlag.APPEND, CreateFlag.NEW_BLOCK);
FSDataOutputStream out = ((DistributedFileSystem) fs).append(path, 128 * 1024, null, flags);

try {
  out = dfs.append(path, bufSize, null, EnumSet.of(CreateFlag.APPEND, CreateFlag.NEW_BLOCK));
} catch (UnsupportedOperationException e) {
  if (e.getMessage().contains("NEW_BLOCK")) {
    // caller did not send NEW_BLOCK on an EC file: fix flags, never retry unchanged
  }
  throw e;
}

When it happens

Trigger: DistributedFileSystem.append(path) without EnumSet.of(CreateFlag.APPEND, CreateFlag.NEW_BLOCK) on a file laid out with an EC policy (directory covered by 'hdfs ec -enablePolicy' / '-setErasureCodingPolicy', or file created with an EC policy).

Common situations: Enabling EC on directory trees that existing generic append-based writers then target; Hadoop 2 to 3 migrations where EC zones appear around append-heavy workloads; frameworks (Spark/Hive appenders, Flume HDFS sink) that do not send NEW_BLOCK.

Related errors

• Non existing file: ${path}. Create option is not specified i
• ${flag} does not contain APPEND
• FileSystem ${item.fs.getUri()} does not support Erasure Codi
• ${message}: ${currentFilePath} [${t}]
• All negative block group IDs are used, growing into positive

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a275bdd905b27417. Report an issue: GitHub.