{"record":{"id":"2f0c92f9a1ed4ff2","repo":"apache/hadoop","slug":"failed-to-recover-the-block-cachefile-in-persis","errorCode":null,"errorMessage":"Failed to recover the block ${cacheFile} in persistent storage.","messagePattern":"Failed to recover the block (.+?) in persistent storage\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/NativePmemMappableBlockLoader.java","lineNumber":193,"sourceCode":"      if (region != null) {\n        POSIX.Pmem.memSync(region);\n      }\n    }\n  }\n\n  @Override\n  public boolean isNativeLoader() {\n    return true;\n  }\n\n  @Override\n  public MappableBlock getRecoveredMappableBlock(\n      File cacheFile, String bpid, byte volumeIndex) throws IOException {\n    NativeIO.POSIX.PmemMappedRegion region =\n        NativeIO.POSIX.Pmem.mapBlock(cacheFile.getAbsolutePath(),\n            cacheFile.length(), true);\n    if (region == null) {\n      throw new IOException(\"Failed to recover the block \"\n          + cacheFile.getName() + \" in persistent storage.\");\n    }\n    ExtendedBlockId key =\n        new ExtendedBlockId(super.getBlockId(cacheFile), bpid);\n    MappableBlock mappableBlock = new NativePmemMappedBlock(\n        region.getAddress(), region.getLength(), key);\n    PmemVolumeManager.getInstance().recoverBlockKeyToVolume(key, volumeIndex);\n\n    String path = PmemVolumeManager.getInstance().getCachePath(key);\n    long addr = mappableBlock.getAddress();\n    long length = mappableBlock.getLength();\n    LOG.info(\"Recovering persistent memory cache for block {}, \" +\n        \"path = {}, address = {}, length = {}\", key, path, addr, length);\n    return mappableBlock;\n  }\n}\n","sourceCodeStart":175,"sourceCodeEnd":210,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/NativePmemMappableBlockLoader.java#L175-L210","documentation":"Thrown by NativePmemMappableBlockLoader.getRecoveredMappableBlock() (NativePmemMappableBlockLoader.java:193) during DataNode restart with dfs.datanode.pmem.cache.recovery=true: NativeIO.POSIX.Pmem.mapBlock(cacheFile.getAbsolutePath(), len, true) returned null, so a cache file found under the pmem volume could not be re-mapped into the address space. Recovery skips or aborts for that block depending on the caller, leaving that replica uncached.","triggerScenarios":"Restart with pmem cache recovery enabled while the pmem volume is missing/reformatted/not mounted; cache file under hdfs_pmem_cache deleted, moved, zero-length, or unreadable by the DataNode user; libpmem failing to map the recovered file (corrupt pmem state).","commonSituations":"pmem mounts not restored before DN start after maintenance; pmem device reformatted (devdax/fsdax reconfiguration) making old cache files stale; ownership of cache files changed when the DN user changed; pmem fill-state changed so mapping the previous length fails.","solutions":["Confirm the pmem volume is mounted exactly as before (same path, fsdax) and the hdfs_pmem_cache tree is intact and owned by the DataNode user","If the old cache content is disposable, clear it (rm -rf <pmem>/hdfs_pmem_cache) - blocks simply re-cache on demand from disk","Verify libpmem presence with hadoop checknative and remount DAX if the device was reconfigured","If recovery is not needed, set dfs.datanode.pmem.cache.recovery=false and restart the DataNode"],"exampleFix":"# before: recovery fails on stale cache files from a reformatted device\n# (ERROR 'Failed to recover the block blk_... in persistent storage.')\n\n# after: reset cache state cleanly and let blocks re-cache lazily\nrm -rf /pmem0/hdfs_pmem_cache/BP-*/\nhdfs --daemon stop datanode && hdfs --daemon start datanode","handlingStrategy":"validation","validationCode":"// before enabling cache recovery, confirm the pmem cache tree still matches the mounts\nfor (String dir : pmemDirs) {\n  File cacheRoot = new File(dir, \"hdfs_pmem_cache\");\n  if (!cacheRoot.isDirectory()\n      || !java.nio.file.Files.isReadable(cacheRoot.toPath())) {\n    throw new IOException(\"pmem cache root unreadable: \" + cacheRoot\n        + \" - fix mounts/ownership or disable dfs.datanode.pmem.cache.recovery\");\n  }\n}","typeGuard":null,"tryCatchPattern":"// recovery loop: skip unrecoverable blocks instead of aborting DN startup\ntry {\n  MappableBlock blk = loader.getRecoveredMappableBlock(cacheFile, bpid, volumeIndex);\n} catch (IOException e) {\n  LOG.warn(\"Could not recover pmem cache file {}: {}\", cacheFile, e);\n  cacheFile.delete(); // block will re-cache from disk on demand\n}","preventionTips":["Keep pmem mount paths stable across maintenance; mount them before starting DataNodes","If the pmem device was reformatted, clear hdfs_pmem_cache rather than attempting recovery","Maintain the same DataNode user so cache file ownership stays valid"],"tags":["hdfs","datanode","pmem","cache-recovery","native-io"],"backgroundTag":"memory-map-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}