{"record":{"id":"f9b1f291671af2fa","repo":"apache/hadoop","slug":"failed-to-map-the-block-blockfilename-to-persis","errorCode":null,"errorMessage":"Failed to map the block ${blockFileName} to persistent storage.","messagePattern":"Failed to map the block (.+?) to persistent storage\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/NativePmemMappableBlockLoader.java","lineNumber":94,"sourceCode":"  @Override\n  public MappableBlock load(long length, FileInputStream blockIn,\n      FileInputStream metaIn, String blockFileName,\n      ExtendedBlockId key)\n      throws IOException {\n    NativePmemMappedBlock mappableBlock = null;\n    POSIX.PmemMappedRegion region = null;\n    String filePath = null;\n\n    try (FileChannel blockChannel = blockIn.getChannel()) {\n      if (blockChannel == null) {\n        throw new IOException(\"Block InputStream has no FileChannel.\");\n      }\n\n      assert NativeIO.isAvailable();\n      filePath = PmemVolumeManager.getInstance().getCachePath(key);\n      region = POSIX.Pmem.mapBlock(filePath, length, false);\n      if (region == null) {\n        throw new IOException(\"Failed to map the block \" + blockFileName +\n            \" to persistent storage.\");\n      }\n      verifyChecksumAndMapBlock(region, length, metaIn, blockChannel,\n          blockFileName);\n      mappableBlock = new NativePmemMappedBlock(region.getAddress(),\n          region.getLength(), key);\n      LOG.info(\"Successfully cached one replica:{} into persistent memory\"\n              + \", [cached path={}, address={}, length={}]\", key, filePath,\n          region.getAddress(), length);\n    } finally {\n      if (mappableBlock == null) {\n        if (region != null) {\n          // unmap content from persistent memory\n          POSIX.Pmem.unmapBlock(region.getAddress(),\n              region.getLength());\n          FsDatasetUtil.deleteMappedFile(filePath);\n        }\n      }","sourceCodeStart":76,"sourceCodeEnd":112,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/NativePmemMappableBlockLoader.java#L76-L112","documentation":"Thrown by NativePmemMappableBlockLoader.load() (NativePmemMappableBlockLoader.java:94) when NativeIO.POSIX.Pmem.mapBlock(filePath, length, false) returns null, i.e. the native pmem_map_file call failed and the block could not be mapped into persistent memory for caching. The target path comes from PmemVolumeManager.getCachePath(key) based on dfs.datanode.pmem.cache.dirs, so common root causes are missing/unhealthy pmem volumes, no space left on them, or libpmem not functioning.","triggerScenarios":"hdfs cacheadmin directive with dfs.datanode.pmem.cache.dirs pointing at a directory that is no longer a valid fsdax mount; pmem capacity exhausted (volume full); libpmem not present/loaded so the native call fails; cache path parent dirs could not be created under the pmem volume.","commonSituations":"Enabling pmem caching before verifying pmem mounts (should be mounted -o dax); pmem device filled by prior cache files after recovery failed to clean them; package libpmem missing so NativeIO.isAvailable is true but Pmem calls fail; pmem volume reformatted/relabeled between DN restarts.","solutions":["Verify dfs.datanode.pmem.cache.dirs points to real pmem directories mounted with DAX (mount | grep dax) and that the DataNode user can write them","Check free space on the pmem volumes (df -h) and clear stale cache content under hdfs_pmem_cache if recovery leftovers filled them","Confirm libpmem is installed and Hadoop's native library picks it up (hadoop checknative -a | grep pmem)","Remount/recreate the fsdax filesystem if it is inconsistent, then restart the DataNode","If pmem caching is not required, remove dfs.datanode.pmem.cache.dirs so the cache falls back to the DRAM loader"],"exampleFix":"# before: pmem dirs stale/unwritable, every cache attempt throws\n<property>\n  <name>dfs.datanode.pmem.cache.dirs</name>\n  <value>/pmem0</value>\n</property>\n\n# after: verified fsdax mount with writable cache root\nmkfs.ext4 -b 4096 /dev/pmem0 && mount -o dax /dev/pmem0 /pmem0\nmkdir -p /pmem0/hdfs_pmem_cache && chown hdfs:hdfs /pmem0/hdfs_pmem_cache","handlingStrategy":"validation","validationCode":"// before enabling pmem caching: dirs exist, are DAX-mounted, writable, and have capacity\nfor (String dir : conf.getTrimmedStrings(\"dfs.datanode.pmem.cache.dirs\")) {\n  File f = new File(dir);\n  if (!f.isDirectory() || !f.canWrite()) {\n    throw new IOException(\"pmem dir invalid: \" + dir);\n  }\n  if (f.getUsableSpace() < minPmemBytes) {\n    throw new IOException(\"pmem dir nearly full: \" + dir);\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  region = POSIX.Pmem.mapBlock(filePath, length, false);\n} catch (IOException e) {\n  // mapBlock null/failure: verify pmem mounts, space, libpmem; skip caching this block and retry later\n  LOG.warn(\"pmem map failed for {} ({}): {}\", blockFileName, filePath, e);\n}","preventionTips":["Mount pmem namespaces with -o dax (fsdax) and verify with mount before starting the DN","Run hadoop checknative -a and confirm the pmem native bits are present","Alert on pmem capacity; clean hdfs_pmem_cache leftovers after failed recoveries","Keep the DataNode user owning the pmem cache root directory"],"tags":["hdfs","datanode","pmem","centralized-cache","native-io","mmap"],"backgroundTag":"memory-map-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}