apache/hadoop · error · IOException

LazyWriter fail to find or create lazy persist dir: ${lazyPe

Error message

LazyWriter fail to find or create lazy persist dir: ${lazyPersistDir}

What it means

Thrown by FsVolumeImpl.copyBlockToLazyPersistLocation() (FsVolumeImpl.java:1578) when the destination directory for lazy persistence (bpDir/current/lazy-persist) does not exist and mkdirs() fails. This runs on the RamDiskAsyncLazyPersistService thread when a LAZY_PERSIST replica must be evicted from RAM_DISK to disk, so failure means the copy is skipped and, until it succeeds, that replica exists only in RAM - lost if the DataNode restarts.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java:1578

        .setGenerationStamp(replicaInfo.getGenerationStamp())
        .setFsVolume(this)
        .setDirectoryToUse(blockFiles[0].getParentFile())
        .setBytesToReserve(0)
        .build();
    newReplicaInfo.setNumBytes(blockFiles[1].length());
    return newReplicaInfo;
  }

  public File[] copyBlockToLazyPersistLocation(String bpId, long blockId,
      long genStamp,
      ReplicaInfo replicaInfo,
      int smallBufferSize,
      Configuration conf) throws IOException {

    File lazyPersistDir  = getLazyPersistDir(bpId);
    if (!lazyPersistDir.exists() && !lazyPersistDir.mkdirs()) {
      FsDatasetImpl.LOG.warn("LazyWriter failed to create " + lazyPersistDir);
      throw new IOException("LazyWriter fail to find or " +
          "create lazy persist dir: " + lazyPersistDir.toString());
    }

    // No FsDatasetImpl lock for the file copy
    File[] targetFiles = FsDatasetImpl.copyBlockFiles(
        blockId, genStamp, replicaInfo, lazyPersistDir, true,
        smallBufferSize, conf);
    return targetFiles;
  }

  public void incrNumBlocks(String bpid) throws IOException {
    getBlockPoolSlice(bpid).incrNumBlocks();
  }

  public void resolveDuplicateReplicas(String bpid, ReplicaInfo memBlockInfo,
      ReplicaInfo diskBlockInfo, ReplicaMap volumeMap) throws IOException {
    getBlockPoolSlice(bpid).resolveDuplicateReplicas(
        memBlockInfo, diskBlockInfo, volumeMap);

View on GitHub (pinned to 2add963021)

Solutions

Verify the DataNode user can create dirs under the volume's bpDir/current (sudo -u hdfs mkdir .../lazy-persist) and fix ownership on data dirs
Ensure at least one writable DISK volume exists alongside RAM_DISK for lazy persist targets
Free space on the target volume or reduce dfs.datanode.du.reserved
If lazy persist is not actually wanted, stop writing LAZY_PERSIST files instead of leaving eviction failing in a loop
Check for a stray regular file named lazy-persist blocking mkdirs and remove it

Example fix

# before: mkdirs fails silently each eviction cycle; replicas stay RAM-only
# (WARN 'LazyWriter failed to create /data/dfs/dn/current/BP-.../current/lazy-persist')

# after: pre-create with correct ownership at volume setup time
install -d -o hdfs -g hdfs /data/dfs/dn/current/BP-*/current/lazy-persist
chmod 755 /data/dfs/dn/current/BP-*/current/lazy-persist

Defensive patterns

Strategy: validation

Validate before calling

// at volume setup or DN start, confirm the lazy-persist dir can be created
File lazy = new File(bpCurrentDir, "lazy-persist");
if (!lazy.exists() && !lazy.mkdirs() && !lazy.exists()) {
  throw new IOException("Cannot create lazy-persist dir " + lazy
      + " - check ownership/space/read-only mount");
}

Try / catch

// inside a lazy-persist eviction task: catch, keep the replica scheduled, do not drop it silently
try {
  volume.copyBlockToLazyPersistLocation(bpid, blockId, genStamp, replica, bufSize, conf);
} catch (IOException e) {
  LOG.warn("Lazy persist of {} failed ({}); retry scheduled", replica, e);
  // re-queue the replica so it is not lost on restart while RAM-only
}

Prevention

Provision data dirs owned by the DataNode user with install -d at setup time
Always pair RAM_DISK volumes with at least one writable DISK volume when using LAZY_PERSIST
Watch for the companion WARN 'LazyWriter failed to create' - it precedes this IOException

When it happens

Trigger: Lazy persist enabled (dfs.datanode.lazywriter.interval.sec configured / storage type LAZY_PERSIST) while the target volume is read-only, full, or not writable by the DataNode user; the lazy-persist path exists as a regular file so mkdirs() returns false; SELinux/AppArmor denying mkdir.

Common situations: Volumes whose permissions changed after DN start or after provisioning as root; no DISK volume configured (only RAM_DISK) so the lazy writer has nowhere to persist; disk full because dfs.datanode.du.reserved consumes everything; hardened hosts blocking writes under the data dirs.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4214afe70f55a980. Report an issue: GitHub.