{"record":{"id":"f9446f84a6122eee","repo":"apache/hadoop","slug":"failed-to-unmap-the-mapped-file-from-pmem-address","errorCode":null,"errorMessage":"Failed to unmap the mapped file from pmem address: ${pmemMappedAddress}","messagePattern":"Failed to unmap the mapped file from pmem address: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"warning","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/NativePmemMappedBlock.java","lineNumber":78,"sourceCode":"\n  @Override\n  public ExtendedBlockId getKey() {\n    return key;\n  }\n\n  @Override\n  public void close() {\n    if (pmemMappedAddress != -1L) {\n      try {\n        String cacheFilePath =\n            PmemVolumeManager.getInstance().getCachePath(key);\n        // Current libpmem will report error when pmem_unmap is called with\n        // length not aligned with page size, although the length is returned\n        // by pmem_map_file.\n        boolean success =\n            NativeIO.POSIX.Pmem.unmapBlock(pmemMappedAddress, length);\n        if (!success) {\n          throw new IOException(\"Failed to unmap the mapped file from \" +\n              \"pmem address: \" + pmemMappedAddress);\n        }\n        pmemMappedAddress = -1L;\n        FsDatasetUtil.deleteMappedFile(cacheFilePath);\n        LOG.info(\"Successfully uncached one replica:{} from persistent memory\"\n            + \", [cached path={}, length={}]\", key, cacheFilePath, length);\n      } catch (IOException e) {\n        LOG.warn(\"IOException occurred for block {}!\", key, e);\n      }\n    }\n  }\n}\n","sourceCodeStart":60,"sourceCodeEnd":91,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/NativePmemMappedBlock.java#L60-L91","documentation":"Constructed in NativePmemMappedBlock.close() (NativePmemMappedBlock.java:78) when NativeIO.POSIX.Pmem.unmapBlock(pmemMappedAddress, length) reports failure during uncache. Unlike the other errors here it is immediately caught and downgraded to LOG.warn ('IOException occurred for block ...!'), so it never propagates - but the mapping stays live and the cache file is not deleted, leaking pmem until process exit. The code comment notes libpmem errors when pmem_unmap gets a length not page-aligned even though pmem_map_file returned it.","triggerScenarios":"Uncaching a pmem replica whose mapped length is not page-size aligned on a libpmem version that rejects it; double-close racing (address already -1 guards this, but concurrent closes can interleave); pmem device state disturbed underneath the process (device removed, driver reset).","commonSituations":"Blocks whose length is not a multiple of the page size combined with older libpmem; frequent cache/uncache cycles where a WARN occasionally appears during dfsadmin cache management; node-level pmem reconfiguration while the DataNode runs.","solutions":["Note it is non-fatal (warn + possible mapping leak until restart); no client operation fails because of it","Upgrade libpmem/Hadoop natives to pick up page-alignment handling in unmap, then restart the DataNode to clear leaked mappings","Monitor pmem usage (ndctl / free) after long cache churn; restart the DN during a maintenance window if usage creeps from leaked mappings","If uncache storms trigger it, space out cache directive removals"],"exampleFix":null,"handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"// the class already swallows and warns; callers of close() need no handler.\n// For tooling, watch the log pattern rather than catching:\n//   LOG.warn(\"IOException occurred for block {}!\", key, e) in NativePmemMappedBlock.close()\nif (pmemUsageCreepsUpWithoutCacheEntries()) {\n  scheduleDatanodeRestart(maintenanceWindow); // clears leaked mappings\n}","preventionTips":["Keep libpmem and Hadoop natives current so page-aligned unmap works","Track pmem usage (ndctl/numastat) vs cached-block count; divergence signals leaked mappings","Restart DataNodes during maintenance windows after heavy cache churn to reclaim any leaked pmem","Remember the WARN is non-fatal - do not page on it, but do trend it"],"tags":["hdfs","datanode","pmem","munmap","resource-leak"],"backgroundTag":"memory-unmap-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}