apache/hadoop · error · IOException

Failed to recover the block ${cacheFile} in persistent stora

Error message

Failed to recover the block ${cacheFile} in persistent storage.

What it means

Thrown by NativePmemMappableBlockLoader.getRecoveredMappableBlock() (NativePmemMappableBlockLoader.java:193) during DataNode restart with dfs.datanode.pmem.cache.recovery=true: NativeIO.POSIX.Pmem.mapBlock(cacheFile.getAbsolutePath(), len, true) returned null, so a cache file found under the pmem volume could not be re-mapped into the address space. Recovery skips or aborts for that block depending on the caller, leaving that replica uncached.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/NativePmemMappableBlockLoader.java:193

      if (region != null) {
        POSIX.Pmem.memSync(region);
      }
    }
  }

  @Override
  public boolean isNativeLoader() {
    return true;
  }

  @Override
  public MappableBlock getRecoveredMappableBlock(
      File cacheFile, String bpid, byte volumeIndex) throws IOException {
    NativeIO.POSIX.PmemMappedRegion region =
        NativeIO.POSIX.Pmem.mapBlock(cacheFile.getAbsolutePath(),
            cacheFile.length(), true);
    if (region == null) {
      throw new IOException("Failed to recover the block "
          + cacheFile.getName() + " in persistent storage.");
    }
    ExtendedBlockId key =
        new ExtendedBlockId(super.getBlockId(cacheFile), bpid);
    MappableBlock mappableBlock = new NativePmemMappedBlock(
        region.getAddress(), region.getLength(), key);
    PmemVolumeManager.getInstance().recoverBlockKeyToVolume(key, volumeIndex);

    String path = PmemVolumeManager.getInstance().getCachePath(key);
    long addr = mappableBlock.getAddress();
    long length = mappableBlock.getLength();
    LOG.info("Recovering persistent memory cache for block {}, " +
        "path = {}, address = {}, length = {}", key, path, addr, length);
    return mappableBlock;
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. Confirm the pmem volume is mounted exactly as before (same path, fsdax) and the hdfs_pmem_cache tree is intact and owned by the DataNode user
2. If the old cache content is disposable, clear it (rm -rf <pmem>/hdfs_pmem_cache) - blocks simply re-cache on demand from disk
3. Verify libpmem presence with hadoop checknative and remount DAX if the device was reconfigured
4. If recovery is not needed, set dfs.datanode.pmem.cache.recovery=false and restart the DataNode

Example fix

# before: recovery fails on stale cache files from a reformatted device
# (ERROR 'Failed to recover the block blk_... in persistent storage.')

# after: reset cache state cleanly and let blocks re-cache lazily
rm -rf /pmem0/hdfs_pmem_cache/BP-*/
hdfs --daemon stop datanode && hdfs --daemon start datanode

// before enabling cache recovery, confirm the pmem cache tree still matches the mounts
for (String dir : pmemDirs) {
  File cacheRoot = new File(dir, "hdfs_pmem_cache");
  if (!cacheRoot.isDirectory()
      || !java.nio.file.Files.isReadable(cacheRoot.toPath())) {
    throw new IOException("pmem cache root unreadable: " + cacheRoot
        + " - fix mounts/ownership or disable dfs.datanode.pmem.cache.recovery");
  }
}

// recovery loop: skip unrecoverable blocks instead of aborting DN startup
try {
  MappableBlock blk = loader.getRecoveredMappableBlock(cacheFile, bpid, volumeIndex);
} catch (IOException e) {
  LOG.warn("Could not recover pmem cache file {}: {}", cacheFile, e);
  cacheFile.delete(); // block will re-cache from disk on demand
}

When it happens

Trigger: Restart with pmem cache recovery enabled while the pmem volume is missing/reformatted/not mounted; cache file under hdfs_pmem_cache deleted, moved, zero-length, or unreadable by the DataNode user; libpmem failing to map the recovered file (corrupt pmem state).

Common situations: pmem mounts not restored before DN start after maintenance; pmem device reformatted (devdax/fsdax reconfiguration) making old cache files stale; ownership of cache files changed when the DN user changed; pmem fill-state changed so mapping the previous length fails.

Related errors

• Failed to map the block ${blockFileName} to persistent stora
• Block InputStream has no FileChannel.
• Cannot get FileChannel from Block InputStream meta file.
• Checksum verification failed for the block ${blockFileName}:
• Failed to unmap the mapped file from pmem address: ${pmemMap

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/2f0c92f9a1ed4ff2. Report an issue: GitHub.