apache/hadoop · critical · FileNotFoundException

{blockURI}

Error message

{blockURI}

What it means

FileNotFoundException thrown by FsDatasetImpl.checkBlock when the volumeMap contains the replica but its data file no longer exists on the filesystem (blockDataExists() is false; the thrown message is the block's URI). This is the classic signature of on-disk data loss or external interference with data directories: HDFS bookkeeping and the filesystem disagree.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:2263

   *                                              was an error locating it.
   * @throws EOFException                      If the replica length is too short.
   * 
   * @throws IOException                       May be thrown from the methods called. 
   */
  @Override // FsDatasetSpi
  public void checkBlock(ExtendedBlock b, long minLength, ReplicaState state)
      throws ReplicaNotFoundException, UnexpectedReplicaStateException,
      FileNotFoundException, EOFException, IOException {
    final ReplicaInfo replicaInfo = volumeMap.get(b.getBlockPoolId(), 
        b.getLocalBlock());
    if (replicaInfo == null) {
      throw new ReplicaNotFoundException(b);
    }
    if (replicaInfo.getState() != state) {
      throw new UnexpectedReplicaStateException(b,state);
    }
    if (!replicaInfo.blockDataExists()) {
      throw new FileNotFoundException(replicaInfo.getBlockURI().toString());
    }
    long onDiskLength = getLength(b);
    if (onDiskLength < minLength) {
      throw new EOFException(b + "'s on-disk length " + onDiskLength
          + " is shorter than minLength " + minLength);
    }
  }

  /**
   * Check whether the given block is a valid one.
   * valid means finalized
   */
  @Override // FsDatasetSpi
  public boolean isValidBlock(ExtendedBlock b) {
    // If block passed is null, we should return false.
    if (b == null) {
      return false;
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Run hdfs fsck on affected paths to enumerate lost replicas and trigger re-replication.
2. Audit what deleted the files (outside processes must never touch data.dir; mount it read-only for non-HDFS users).
3. Check dmesg/smartctl for hardware errors; replace the failing disk and reformat the volume via DataNode re-registration.
4. Verify data directories are on persistent storage in containerized deployments (no emptyDir for data).

boolean isMissingBlockFile(IOException e) {
  return e instanceof java.io.FileNotFoundException
      && e.getMessage() != null && e.getMessage().startsWith("file:"); // block URI
}

try {
  dataset.checkBlock(b, minLength, ReplicaState.FINALIZED);
} catch (FileNotFoundException e) {
  // on-disk loss: fall back to another replica AND raise a disk-integrity alert
  diskIntegrityAlert(b, e);
  return nextReplica(b);
}

When it happens

Trigger: checkBlock(b, minLength, state) on a replica whose block file was deleted outside HDFS or lost - the volumeMap entry survives but replicaInfo.blockDataExists() returns false.

Common situations: Disk failure or corrupted filesystem; an operator/Nagios/backup job deleting files under dfs.datanode.data.dir; container ephemeral storage wiped on restart; partial restore from snapshot; fs corruption after power loss.

Related errors

• Block data not found, r={r}
• {} has no enough internal blocks(current: {}), unable to sta
• Found fewer bytesOnDisk than bytesAcked for replica {rbw}
• No volume for bpid: {bpid}, block: {replicaInfo}
• Replica not found for {b}. The block may have been removed r

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/433f2164d10b2a7b. Report an issue: GitHub.