apache/hadoop · error · IOException

Failed to map the block ${blockFileName} to persistent stora

Error message

Failed to map the block ${blockFileName} to persistent storage.

What it means

Thrown by NativePmemMappableBlockLoader.load() (NativePmemMappableBlockLoader.java:94) when NativeIO.POSIX.Pmem.mapBlock(filePath, length, false) returns null, i.e. the native pmem_map_file call failed and the block could not be mapped into persistent memory for caching. The target path comes from PmemVolumeManager.getCachePath(key) based on dfs.datanode.pmem.cache.dirs, so common root causes are missing/unhealthy pmem volumes, no space left on them, or libpmem not functioning.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/NativePmemMappableBlockLoader.java:94

  @Override
  public MappableBlock load(long length, FileInputStream blockIn,
      FileInputStream metaIn, String blockFileName,
      ExtendedBlockId key)
      throws IOException {
    NativePmemMappedBlock mappableBlock = null;
    POSIX.PmemMappedRegion region = null;
    String filePath = null;

    try (FileChannel blockChannel = blockIn.getChannel()) {
      if (blockChannel == null) {
        throw new IOException("Block InputStream has no FileChannel.");
      }

      assert NativeIO.isAvailable();
      filePath = PmemVolumeManager.getInstance().getCachePath(key);
      region = POSIX.Pmem.mapBlock(filePath, length, false);
      if (region == null) {
        throw new IOException("Failed to map the block " + blockFileName +
            " to persistent storage.");
      }
      verifyChecksumAndMapBlock(region, length, metaIn, blockChannel,
          blockFileName);
      mappableBlock = new NativePmemMappedBlock(region.getAddress(),
          region.getLength(), key);
      LOG.info("Successfully cached one replica:{} into persistent memory"
              + ", [cached path={}, address={}, length={}]", key, filePath,
          region.getAddress(), length);
    } finally {
      if (mappableBlock == null) {
        if (region != null) {
          // unmap content from persistent memory
          POSIX.Pmem.unmapBlock(region.getAddress(),
              region.getLength());
          FsDatasetUtil.deleteMappedFile(filePath);
        }
      }

View on GitHub (pinned to 2add963021)

Solutions

1. Verify dfs.datanode.pmem.cache.dirs points to real pmem directories mounted with DAX (mount | grep dax) and that the DataNode user can write them
2. Check free space on the pmem volumes (df -h) and clear stale cache content under hdfs_pmem_cache if recovery leftovers filled them
3. Confirm libpmem is installed and Hadoop's native library picks it up (hadoop checknative -a | grep pmem)
4. Remount/recreate the fsdax filesystem if it is inconsistent, then restart the DataNode
5. If pmem caching is not required, remove dfs.datanode.pmem.cache.dirs so the cache falls back to the DRAM loader

Example fix

# before: pmem dirs stale/unwritable, every cache attempt throws
<property>
  <name>dfs.datanode.pmem.cache.dirs</name>
  <value>/pmem0</value>
</property>

# after: verified fsdax mount with writable cache root
mkfs.ext4 -b 4096 /dev/pmem0 && mount -o dax /dev/pmem0 /pmem0
mkdir -p /pmem0/hdfs_pmem_cache && chown hdfs:hdfs /pmem0/hdfs_pmem_cache

// before enabling pmem caching: dirs exist, are DAX-mounted, writable, and have capacity
for (String dir : conf.getTrimmedStrings("dfs.datanode.pmem.cache.dirs")) {
  File f = new File(dir);
  if (!f.isDirectory() || !f.canWrite()) {
    throw new IOException("pmem dir invalid: " + dir);
  }
  if (f.getUsableSpace() < minPmemBytes) {
    throw new IOException("pmem dir nearly full: " + dir);
  }
}

try {
  region = POSIX.Pmem.mapBlock(filePath, length, false);
} catch (IOException e) {
  // mapBlock null/failure: verify pmem mounts, space, libpmem; skip caching this block and retry later
  LOG.warn("pmem map failed for {} ({}): {}", blockFileName, filePath, e);
}

When it happens

Trigger: hdfs cacheadmin directive with dfs.datanode.pmem.cache.dirs pointing at a directory that is no longer a valid fsdax mount; pmem capacity exhausted (volume full); libpmem not present/loaded so the native call fails; cache path parent dirs could not be created under the pmem volume.

Common situations: Enabling pmem caching before verifying pmem mounts (should be mounted -o dax); pmem device filled by prior cache files after recovery failed to clean them; package libpmem missing so NativeIO.isAvailable is true but Pmem calls fail; pmem volume reformatted/relabeled between DN restarts.

Related errors

• Block InputStream has no FileChannel.
• Cannot get FileChannel from Block InputStream meta file.
• Checksum verification failed for the block ${blockFileName}:
• Failed to recover the block ${cacheFile} in persistent stora
• Block InputStream has no FileChannel.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f9b1f291671af2fa. Report an issue: GitHub.