{"record":{"id":"c80210807c304480","repo":"apache/hadoop","slug":"block-data-not-found-r-r","errorCode":null,"errorMessage":"Block data not found, r={r}","messagePattern":"Block data not found, r=(.+?)","errorType":"exception","errorClass":"FileNotFoundException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":2330,"sourceCode":"    final ReplicaInfo r;\n    r = volumeMap.get(bpid, blockId);\n    if (r != null) {\n      if (r.blockDataExists()) {\n        return r;\n      }\n      // if file is not null, but doesn't exist - possibly disk failed\n      datanode.checkDiskErrorAsync(r.getVolume());\n    }\n\n    LOG.debug(\"blockId={}, replica={}\", blockId, r);\n    return null;\n  }\n\n  /** Check the files of a replica. */\n  static void checkReplicaFiles(final ReplicaInfo r) throws IOException {\n    //check replica's data exists\n    if (!r.blockDataExists()) {\n      throw new FileNotFoundException(\"Block data not found, r=\" + r);\n    }\n    if (r.getBytesOnDisk() != r.getBlockDataLength()) {\n      throw new IOException(\"Block length mismatch, len=\"\n          + r.getBlockDataLength() + \" but r=\" + r);\n    }\n\n    //check replica's meta file\n    if (!r.metadataExists()) {\n      throw new IOException(r.getMetadataURI() + \" does not exist, r=\" + r);\n    }\n    if (r.getMetadataLength() == 0) {\n      throw new IOException(\"Metafile is empty, r=\" + r);\n    }\n  }\n\n  /**\n   * We're informed that a block is no longer valid. Delete it.\n   */","sourceCodeStart":2312,"sourceCodeEnd":2348,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L2312-L2348","documentation":"FileNotFoundException thrown by the static FsDatasetImpl.checkReplicaFiles when a replica's block data file does not exist on disk (r.blockDataExists() false; message embeds the whole replica descriptor). checkReplicaFiles is invoked from recovery paths (initReplicaRecoveryImpl) and finalize paths to validate the replica's files before mutating state, so recovery aborts when the data file is gone.","triggerScenarios":"checkReplicaFiles(r) during initReplicaRecovery or finalize on a replica whose data file was deleted or lost while the volumeMap entry remained - the state machine thinks the replica exists, the filesystem disagrees.","commonSituations":"External deletion under dfs.datanode.data.dir; failing disk dropped files; restored-inconsistent volume after crash; replica invalidated by another thread between volumeMap lookup and file check.","solutions":["Run hdfs fsck to confirm replication health and let the NameNode replace the lost replica.","Find what removed the file (operator action, monitoring script, container storage lifecycle).","Check the disk (smartctl/dmesg) and DataNode logs for prior 'checkDiskError' events.","Restart the DataNode to reconcile volumeMap with what is actually on disk."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":"boolean isReplicaDataMissing(IOException e) {\n  return e instanceof java.io.FileNotFoundException\n      && e.getMessage() != null && e.getMessage().startsWith(\"Block data not found\");\n}","tryCatchPattern":"try {\n  ReplicaRecoveryInfo info = dataset.initReplicaRecovery(rBlock);\n} catch (FileNotFoundException e) {\n  // data file gone: skip this DataNode in recovery; other pipeline members carry the block\n  return recoverFromOtherReplicas(rBlock);\n}","preventionTips":["Protect data directories from external modification (monitoring scripts, tmp cleaners).","Alert on any FileNotFoundException from DataNode block paths - bookkeeping/filesystem divergence is always serious.","Verify volume health after any restore or migration of data dirs."],"tags":["hdfs","datanode","replica-files","file-not-found","data-loss"],"backgroundTag":"missing-block-file","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}