apache/hadoop · error · UnsupportedOperationException

"Cannot append to lazy persist file " + path

Error message

"Cannot append to lazy persist file " + path

What it means

Files written with the LAZY_PERSIST storage policy live in memory (RAM_DISK) with lazy flushing to disk, so extending them would add data the NameNode cannot guarantee is persisted. FSDirAppendOp looks up the LAZY_PERSIST policy id, compares it with the file's policy, and throws UnsupportedOperationException before any lease recovery.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java:122

      if (inode == null) {
        throw new FileNotFoundException(
            "Failed to append to non-existent file " + path + " for client "
                + clientMachine);
      }
      final INodeFile file = INodeFile.valueOf(inode, path, true);

      if (file.isStriped() && !newBlock) {
        throw new UnsupportedOperationException(
            "Append on EC file without new block is not supported. Use "
                + CreateFlag.NEW_BLOCK + " create flag while appending file.");
      }

      BlockManager blockManager = fsd.getBlockManager();
      final BlockStoragePolicy lpPolicy = blockManager
          .getStoragePolicy("LAZY_PERSIST");
      if (lpPolicy != null && lpPolicy.getId() == file.getStoragePolicyID()) {
        throw new UnsupportedOperationException(
            "Cannot append to lazy persist file " + path);
      }
      // Opening an existing file for append - may need to recover lease.
      fsn.recoverLeaseInternal(RecoverLeaseOp.APPEND_FILE, iip, path, holder,
          clientMachine, false);

      final BlockInfo lastBlock = file.getLastBlock();
      // Check that the block has at least minimum replication.
      if (lastBlock != null) {
        if (lastBlock.getBlockUCState() == BlockUCState.COMMITTED) {
          throw new RetriableException(
              new NotReplicatedYetException("append: lastBlock="
                  + lastBlock + " of src=" + path
                  + " is COMMITTED but not yet COMPLETE."));
        } else if (lastBlock.isComplete()
          && !blockManager.isSufficientlyReplicated(lastBlock)) {
          throw new IOException("append: lastBlock=" + lastBlock + " of src="
              + path + " is not sufficiently replicated yet.");

View on GitHub (pinned to 2add963021)

Solutions

1. For existing files there is no in-place fix: copy the content into a new HOT-policy file and append to that file (the copy-on-create restriction also blocks changing the file's policy, see the 'cannot be changed after file creation' errors).
2. For future files, scope LAZY_PERSIST to write-once scratch directories only, and never point append-based writers at them.
3. Check before appending: fs.getStoragePolicy(path) returning LAZY_PERSIST means this append will fail - branch to the copy-and-replace path.

Example fix

// before
FSDataOutputStream out = fs.append(path); // UnsupportedOperationException if LAZY_PERSIST

// after: route lazy-persist files through a rewrite
BlockStoragePolicy p = ((DistributedFileSystem) fs).getStoragePolicy(path);
if (p != null && p.isCopyOnCreateFile()) { // LAZY_PERSIST
  Path tmp = new Path(path.getParent(), path.getName() + ".hot");
  FileUtil.copy(fs, path, fs, tmp, false, true, fs.getConf());
  fs.delete(path, false);
  fs.rename(tmp, path);
}
FSDataOutputStream out = fs.append(path);

BlockStoragePolicy p = ((DistributedFileSystem) fs).getStoragePolicy(path);
if (p != null && p.isCopyOnCreateFile()) {
  // LAZY_PERSIST: append unsupported; rewrite into a HOT file first
  Path tmp = new Path(path.getParent(), path.getName() + ".tmp");
  FileUtil.copy(fs, path, fs, tmp, false, true, fs.getConf());
  fs.delete(path, false);
  fs.rename(tmp, path);
}

try {
  out = fs.append(path);
} catch (UnsupportedOperationException e) {
  if (e.getMessage().contains("lazy persist")) {
    // route to a copy-and-replace rewrite instead of retrying
  }
}

When it happens

Trigger: append() on a file whose storage policy is LAZY_PERSIST - created with CreateFlag.LAZY_PERSIST, via fs.create with a lazy-persist builder, or inherited from a directory that had setStoragePolicy(dir, 'LAZY_PERSIST') applied.

Common situations: Using in-memory storage for hot temp data, then trying to extend those files; POC/demo code for RAM_DISK HDFS being reused for ordinary append workloads; policy rollout scripts applying LAZY_PERSIST to whole existing trees that are later appended to.

Related errors

• "Policy " + newPolicy + " cannot be set after file creation.
• "Existing policy " + currentPolicy.getName() + " cannot be c
• The LAZY_PERSIST storage policy has been disabled by the adm
• {} doesn't support satisfyStoragePolicy
• {} doesn't support setStoragePolicy

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/fa51e74af7a41ecf. Report an issue: GitHub.