apache/hadoop · critical · FileNotFoundException

Block data not found, r={r}

Error message

Block data not found, r={r}

What it means

FileNotFoundException thrown by the static FsDatasetImpl.checkReplicaFiles when a replica's block data file does not exist on disk (r.blockDataExists() false; message embeds the whole replica descriptor). checkReplicaFiles is invoked from recovery paths (initReplicaRecoveryImpl) and finalize paths to validate the replica's files before mutating state, so recovery aborts when the data file is gone.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:2330

    final ReplicaInfo r;
    r = volumeMap.get(bpid, blockId);
    if (r != null) {
      if (r.blockDataExists()) {
        return r;
      }
      // if file is not null, but doesn't exist - possibly disk failed
      datanode.checkDiskErrorAsync(r.getVolume());
    }

    LOG.debug("blockId={}, replica={}", blockId, r);
    return null;
  }

  /** Check the files of a replica. */
  static void checkReplicaFiles(final ReplicaInfo r) throws IOException {
    //check replica's data exists
    if (!r.blockDataExists()) {
      throw new FileNotFoundException("Block data not found, r=" + r);
    }
    if (r.getBytesOnDisk() != r.getBlockDataLength()) {
      throw new IOException("Block length mismatch, len="
          + r.getBlockDataLength() + " but r=" + r);
    }

    //check replica's meta file
    if (!r.metadataExists()) {
      throw new IOException(r.getMetadataURI() + " does not exist, r=" + r);
    }
    if (r.getMetadataLength() == 0) {
      throw new IOException("Metafile is empty, r=" + r);
    }
  }

  /**
   * We're informed that a block is no longer valid. Delete it.
   */

View on GitHub (pinned to 2add963021)

Solutions

1. Run hdfs fsck to confirm replication health and let the NameNode replace the lost replica.
2. Find what removed the file (operator action, monitoring script, container storage lifecycle).
3. Check the disk (smartctl/dmesg) and DataNode logs for prior 'checkDiskError' events.
4. Restart the DataNode to reconcile volumeMap with what is actually on disk.

boolean isReplicaDataMissing(IOException e) {
  return e instanceof java.io.FileNotFoundException
      && e.getMessage() != null && e.getMessage().startsWith("Block data not found");
}

try {
  ReplicaRecoveryInfo info = dataset.initReplicaRecovery(rBlock);
} catch (FileNotFoundException e) {
  // data file gone: skip this DataNode in recovery; other pipeline members carry the block
  return recoverFromOtherReplicas(rBlock);
}

When it happens

Trigger: checkReplicaFiles(r) during initReplicaRecovery or finalize on a replica whose data file was deleted or lost while the volumeMap entry remained - the state machine thinks the replica exists, the filesystem disagrees.

Common situations: External deletion under dfs.datanode.data.dir; failing disk dropped files; restored-inconsistent volume after crash; replica invalidated by another thread between volumeMap lookup and file check.

Related errors

• {blockURI}
• {metadataURI} does not exist, r={r}
• {} has no enough internal blocks(current: {}), unable to sta
• Block length mismatch, len={len} but r={r}
• Metafile is empty, r={r}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c80210807c304480. Report an issue: GitHub.