{"record":{"id":"fa51e74af7a41ecf","repo":"apache/hadoop","slug":"cannot-append-to-lazy-persist-file-path","errorCode":null,"errorMessage":"\"Cannot append to lazy persist file \" + path","messagePattern":"\"Cannot append to lazy persist file \" \\+ path","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java","lineNumber":122,"sourceCode":"\n      if (inode == null) {\n        throw new FileNotFoundException(\n            \"Failed to append to non-existent file \" + path + \" for client \"\n                + clientMachine);\n      }\n      final INodeFile file = INodeFile.valueOf(inode, path, true);\n\n      if (file.isStriped() && !newBlock) {\n        throw new UnsupportedOperationException(\n            \"Append on EC file without new block is not supported. Use \"\n                + CreateFlag.NEW_BLOCK + \" create flag while appending file.\");\n      }\n\n      BlockManager blockManager = fsd.getBlockManager();\n      final BlockStoragePolicy lpPolicy = blockManager\n          .getStoragePolicy(\"LAZY_PERSIST\");\n      if (lpPolicy != null && lpPolicy.getId() == file.getStoragePolicyID()) {\n        throw new UnsupportedOperationException(\n            \"Cannot append to lazy persist file \" + path);\n      }\n      // Opening an existing file for append - may need to recover lease.\n      fsn.recoverLeaseInternal(RecoverLeaseOp.APPEND_FILE, iip, path, holder,\n          clientMachine, false);\n\n      final BlockInfo lastBlock = file.getLastBlock();\n      // Check that the block has at least minimum replication.\n      if (lastBlock != null) {\n        if (lastBlock.getBlockUCState() == BlockUCState.COMMITTED) {\n          throw new RetriableException(\n              new NotReplicatedYetException(\"append: lastBlock=\"\n                  + lastBlock + \" of src=\" + path\n                  + \" is COMMITTED but not yet COMPLETE.\"));\n        } else if (lastBlock.isComplete()\n          && !blockManager.isSufficientlyReplicated(lastBlock)) {\n          throw new IOException(\"append: lastBlock=\" + lastBlock + \" of src=\"\n              + path + \" is not sufficiently replicated yet.\");","sourceCodeStart":104,"sourceCodeEnd":140,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java#L104-L140","documentation":"Files written with the LAZY_PERSIST storage policy live in memory (RAM_DISK) with lazy flushing to disk, so extending them would add data the NameNode cannot guarantee is persisted. FSDirAppendOp looks up the LAZY_PERSIST policy id, compares it with the file's policy, and throws UnsupportedOperationException before any lease recovery.","triggerScenarios":"append() on a file whose storage policy is LAZY_PERSIST - created with CreateFlag.LAZY_PERSIST, via fs.create with a lazy-persist builder, or inherited from a directory that had setStoragePolicy(dir, 'LAZY_PERSIST') applied.","commonSituations":"Using in-memory storage for hot temp data, then trying to extend those files; POC/demo code for RAM_DISK HDFS being reused for ordinary append workloads; policy rollout scripts applying LAZY_PERSIST to whole existing trees that are later appended to.","solutions":["For existing files there is no in-place fix: copy the content into a new HOT-policy file and append to that file (the copy-on-create restriction also blocks changing the file's policy, see the 'cannot be changed after file creation' errors).","For future files, scope LAZY_PERSIST to write-once scratch directories only, and never point append-based writers at them.","Check before appending: fs.getStoragePolicy(path) returning LAZY_PERSIST means this append will fail - branch to the copy-and-replace path."],"exampleFix":"// before\nFSDataOutputStream out = fs.append(path); // UnsupportedOperationException if LAZY_PERSIST\n\n// after: route lazy-persist files through a rewrite\nBlockStoragePolicy p = ((DistributedFileSystem) fs).getStoragePolicy(path);\nif (p != null && p.isCopyOnCreateFile()) { // LAZY_PERSIST\n  Path tmp = new Path(path.getParent(), path.getName() + \".hot\");\n  FileUtil.copy(fs, path, fs, tmp, false, true, fs.getConf());\n  fs.delete(path, false);\n  fs.rename(tmp, path);\n}\nFSDataOutputStream out = fs.append(path);","handlingStrategy":"validation","validationCode":"BlockStoragePolicy p = ((DistributedFileSystem) fs).getStoragePolicy(path);\nif (p != null && p.isCopyOnCreateFile()) {\n  // LAZY_PERSIST: append unsupported; rewrite into a HOT file first\n  Path tmp = new Path(path.getParent(), path.getName() + \".tmp\");\n  FileUtil.copy(fs, path, fs, tmp, false, true, fs.getConf());\n  fs.delete(path, false);\n  fs.rename(tmp, path);\n}","typeGuard":null,"tryCatchPattern":"try {\n  out = fs.append(path);\n} catch (UnsupportedOperationException e) {\n  if (e.getMessage().contains(\"lazy persist\")) {\n    // route to a copy-and-replace rewrite instead of retrying\n  }\n}","preventionTips":["Reserve LAZY_PERSIST for write-once scratch data.","Check getStoragePolicy(path).isCopyOnCreateFile() in shared write utilities."],"tags":["hdfs","append","storage-policy","lazy-persist"],"backgroundTag":"append-unsupported-file-type","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}