apache/hadoop · error · IOException

{metadataURI} does not exist, r={r}

Error message

{metadataURI} does not exist, r={r}

What it means

IOException thrown by FsDatasetImpl.checkReplicaFiles when the replica's metadata (checksum) file is missing: metadataExists() returns false, and the message embeds r.getMetadataURI(). Without the .meta checksum file the block cannot be validated or served for reads that verify checksums, so recovery/finalize paths refuse to proceed.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:2339

    LOG.debug("blockId={}, replica={}", blockId, r);
    return null;
  }

  /** Check the files of a replica. */
  static void checkReplicaFiles(final ReplicaInfo r) throws IOException {
    //check replica's data exists
    if (!r.blockDataExists()) {
      throw new FileNotFoundException("Block data not found, r=" + r);
    }
    if (r.getBytesOnDisk() != r.getBlockDataLength()) {
      throw new IOException("Block length mismatch, len="
          + r.getBlockDataLength() + " but r=" + r);
    }

    //check replica's meta file
    if (!r.metadataExists()) {
      throw new IOException(r.getMetadataURI() + " does not exist, r=" + r);
    }
    if (r.getMetadataLength() == 0) {
      throw new IOException("Metafile is empty, r=" + r);
    }
  }

  /**
   * We're informed that a block is no longer valid. Delete it.
   */
  @Override // FsDatasetSpi
  public void invalidate(String bpid, Block invalidBlks[]) throws IOException {
    invalidate(bpid, invalidBlks, true);
  }

  private void invalidate(String bpid, Block[] invalidBlks, boolean async)
      throws IOException {
    final List<String> errors = new ArrayList<String>();
    for (int i = 0; i < invalidBlks.length; i++) {

View on GitHub (pinned to 2add963021)

Solutions

1. Check whether the .meta file path in the message exists elsewhere on the volume (moved by a broken tool) and restore it.
2. Run hdfs fsck -blocks: if other healthy replicas exist, invalidate this one and let re-replication rebuild it.
3. If this was the last replica, hdfs debug recoverLease / restart may allow reading data without checksums as a salvage path.
4. Fix the root cause (crash-consistency, external interference) and restart the DataNode.

boolean isMissingMetaFile(IOException e) {
  return e.getMessage() != null && e.getMessage().endsWith(".meta does not exist") ||
      (e.getMessage() != null && e.getMessage().contains("does not exist, r="));
}

try {
  dataset.initReplicaRecovery(rBlock);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("does not exist, r=")) {
    // missing .meta: exclude this replica from recovery, fsck + re-replicate
    return recoverFromOtherReplicas(rBlock);
  }
  throw e;
}

When it happens

Trigger: checkReplicaFiles(r) during initReplicaRecovery/finalize on a replica whose block file exists but whose .meta file (blockId_generationStamp.meta) was deleted or never written - e.g., crash between data file creation and meta file creation.

Common situations: DataNode crash mid-write before the meta file was flushed; external deletion of *.meta; restore-from-backup that missed meta files; filesystem corruption after power loss.

Related errors

• Block data not found, r={r}
• Metafile is empty, r={r}
• {blockURI}
• Block length mismatch, len={len} but r={r}
• %s does not exist or is not file.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a5cd07cd0b03ceb2. Report an issue: GitHub.