apache/hadoop · warning · IOException

Failed to unmap the mapped file from pmem address: ${pmemMap

Error message

Failed to unmap the mapped file from pmem address: ${pmemMappedAddress}

What it means

Constructed in NativePmemMappedBlock.close() (NativePmemMappedBlock.java:78) when NativeIO.POSIX.Pmem.unmapBlock(pmemMappedAddress, length) reports failure during uncache. Unlike the other errors here it is immediately caught and downgraded to LOG.warn ('IOException occurred for block ...!'), so it never propagates - but the mapping stays live and the cache file is not deleted, leaking pmem until process exit. The code comment notes libpmem errors when pmem_unmap gets a length not page-aligned even though pmem_map_file returned it.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/NativePmemMappedBlock.java:78

  @Override
  public ExtendedBlockId getKey() {
    return key;
  }

  @Override
  public void close() {
    if (pmemMappedAddress != -1L) {
      try {
        String cacheFilePath =
            PmemVolumeManager.getInstance().getCachePath(key);
        // Current libpmem will report error when pmem_unmap is called with
        // length not aligned with page size, although the length is returned
        // by pmem_map_file.
        boolean success =
            NativeIO.POSIX.Pmem.unmapBlock(pmemMappedAddress, length);
        if (!success) {
          throw new IOException("Failed to unmap the mapped file from " +
              "pmem address: " + pmemMappedAddress);
        }
        pmemMappedAddress = -1L;
        FsDatasetUtil.deleteMappedFile(cacheFilePath);
        LOG.info("Successfully uncached one replica:{} from persistent memory"
            + ", [cached path={}, length={}]", key, cacheFilePath, length);
      } catch (IOException e) {
        LOG.warn("IOException occurred for block {}!", key, e);
      }
    }
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. Note it is non-fatal (warn + possible mapping leak until restart); no client operation fails because of it
2. Upgrade libpmem/Hadoop natives to pick up page-alignment handling in unmap, then restart the DataNode to clear leaked mappings
3. Monitor pmem usage (ndctl / free) after long cache churn; restart the DN during a maintenance window if usage creeps from leaked mappings
4. If uncache storms trigger it, space out cache directive removals

// the class already swallows and warns; callers of close() need no handler.
// For tooling, watch the log pattern rather than catching:
//   LOG.warn("IOException occurred for block {}!", key, e) in NativePmemMappedBlock.close()
if (pmemUsageCreepsUpWithoutCacheEntries()) {
  scheduleDatanodeRestart(maintenanceWindow); // clears leaked mappings
}

When it happens

Trigger: Uncaching a pmem replica whose mapped length is not page-size aligned on a libpmem version that rejects it; double-close racing (address already -1 guards this, but concurrent closes can interleave); pmem device state disturbed underneath the process (device removed, driver reset).

Common situations: Blocks whose length is not a multiple of the page size combined with older libpmem; frequent cache/uncache cycles where a WARN occasionally appears during dfsadmin cache management; node-level pmem reconfiguration while the DataNode runs.

Related errors

• Block InputStream has no FileChannel.
• Failed to map the block ${blockFileName} to persistent stora
• Cannot get FileChannel from Block InputStream meta file.
• Checksum verification failed for the block ${blockFileName}:
• Failed to recover the block ${cacheFile} in persistent stora

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f9446f84a6122eee. Report an issue: GitHub.