{"record":{"id":"4214afe70f55a980","repo":"apache/hadoop","slug":"lazywriter-fail-to-find-or-create-lazy-persist-dir","errorCode":null,"errorMessage":"LazyWriter fail to find or create lazy persist dir: ${lazyPersistDir}","messagePattern":"LazyWriter fail to find or create lazy persist dir: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java","lineNumber":1578,"sourceCode":"        .setGenerationStamp(replicaInfo.getGenerationStamp())\n        .setFsVolume(this)\n        .setDirectoryToUse(blockFiles[0].getParentFile())\n        .setBytesToReserve(0)\n        .build();\n    newReplicaInfo.setNumBytes(blockFiles[1].length());\n    return newReplicaInfo;\n  }\n\n  public File[] copyBlockToLazyPersistLocation(String bpId, long blockId,\n      long genStamp,\n      ReplicaInfo replicaInfo,\n      int smallBufferSize,\n      Configuration conf) throws IOException {\n\n    File lazyPersistDir  = getLazyPersistDir(bpId);\n    if (!lazyPersistDir.exists() && !lazyPersistDir.mkdirs()) {\n      FsDatasetImpl.LOG.warn(\"LazyWriter failed to create \" + lazyPersistDir);\n      throw new IOException(\"LazyWriter fail to find or \" +\n          \"create lazy persist dir: \" + lazyPersistDir.toString());\n    }\n\n    // No FsDatasetImpl lock for the file copy\n    File[] targetFiles = FsDatasetImpl.copyBlockFiles(\n        blockId, genStamp, replicaInfo, lazyPersistDir, true,\n        smallBufferSize, conf);\n    return targetFiles;\n  }\n\n  public void incrNumBlocks(String bpid) throws IOException {\n    getBlockPoolSlice(bpid).incrNumBlocks();\n  }\n\n  public void resolveDuplicateReplicas(String bpid, ReplicaInfo memBlockInfo,\n      ReplicaInfo diskBlockInfo, ReplicaMap volumeMap) throws IOException {\n    getBlockPoolSlice(bpid).resolveDuplicateReplicas(\n        memBlockInfo, diskBlockInfo, volumeMap);","sourceCodeStart":1560,"sourceCodeEnd":1596,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java#L1560-L1596","documentation":"Thrown by FsVolumeImpl.copyBlockToLazyPersistLocation() (FsVolumeImpl.java:1578) when the destination directory for lazy persistence (bpDir/current/lazy-persist) does not exist and mkdirs() fails. This runs on the RamDiskAsyncLazyPersistService thread when a LAZY_PERSIST replica must be evicted from RAM_DISK to disk, so failure means the copy is skipped and, until it succeeds, that replica exists only in RAM - lost if the DataNode restarts.","triggerScenarios":"Lazy persist enabled (dfs.datanode.lazywriter.interval.sec configured / storage type LAZY_PERSIST) while the target volume is read-only, full, or not writable by the DataNode user; the lazy-persist path exists as a regular file so mkdirs() returns false; SELinux/AppArmor denying mkdir.","commonSituations":"Volumes whose permissions changed after DN start or after provisioning as root; no DISK volume configured (only RAM_DISK) so the lazy writer has nowhere to persist; disk full because dfs.datanode.du.reserved consumes everything; hardened hosts blocking writes under the data dirs.","solutions":["Verify the DataNode user can create dirs under the volume's bpDir/current (sudo -u hdfs mkdir .../lazy-persist) and fix ownership on data dirs","Ensure at least one writable DISK volume exists alongside RAM_DISK for lazy persist targets","Free space on the target volume or reduce dfs.datanode.du.reserved","If lazy persist is not actually wanted, stop writing LAZY_PERSIST files instead of leaving eviction failing in a loop","Check for a stray regular file named lazy-persist blocking mkdirs and remove it"],"exampleFix":"# before: mkdirs fails silently each eviction cycle; replicas stay RAM-only\n# (WARN 'LazyWriter failed to create /data/dfs/dn/current/BP-.../current/lazy-persist')\n\n# after: pre-create with correct ownership at volume setup time\ninstall -d -o hdfs -g hdfs /data/dfs/dn/current/BP-*/current/lazy-persist\nchmod 755 /data/dfs/dn/current/BP-*/current/lazy-persist","handlingStrategy":"validation","validationCode":"// at volume setup or DN start, confirm the lazy-persist dir can be created\nFile lazy = new File(bpCurrentDir, \"lazy-persist\");\nif (!lazy.exists() && !lazy.mkdirs() && !lazy.exists()) {\n  throw new IOException(\"Cannot create lazy-persist dir \" + lazy\n      + \" - check ownership/space/read-only mount\");\n}","typeGuard":null,"tryCatchPattern":"// inside a lazy-persist eviction task: catch, keep the replica scheduled, do not drop it silently\ntry {\n  volume.copyBlockToLazyPersistLocation(bpid, blockId, genStamp, replica, bufSize, conf);\n} catch (IOException e) {\n  LOG.warn(\"Lazy persist of {} failed ({}); retry scheduled\", replica, e);\n  // re-queue the replica so it is not lost on restart while RAM-only\n}","preventionTips":["Provision data dirs owned by the DataNode user with install -d at setup time","Always pair RAM_DISK volumes with at least one writable DISK volume when using LAZY_PERSIST","Watch for the companion WARN 'LazyWriter failed to create' - it precedes this IOException"],"tags":["hdfs","datanode","lazy-persist","ram-disk","mkdir"],"backgroundTag":"mkdir-permission-denied","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}