{"record":{"id":"a5cd07cd0b03ceb2","repo":"apache/hadoop","slug":"metadatauri-does-not-exist-r-r","errorCode":null,"errorMessage":"{metadataURI} does not exist, r={r}","messagePattern":"(.+?) does not exist, r=(.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":2339,"sourceCode":"\n    LOG.debug(\"blockId={}, replica={}\", blockId, r);\n    return null;\n  }\n\n  /** Check the files of a replica. */\n  static void checkReplicaFiles(final ReplicaInfo r) throws IOException {\n    //check replica's data exists\n    if (!r.blockDataExists()) {\n      throw new FileNotFoundException(\"Block data not found, r=\" + r);\n    }\n    if (r.getBytesOnDisk() != r.getBlockDataLength()) {\n      throw new IOException(\"Block length mismatch, len=\"\n          + r.getBlockDataLength() + \" but r=\" + r);\n    }\n\n    //check replica's meta file\n    if (!r.metadataExists()) {\n      throw new IOException(r.getMetadataURI() + \" does not exist, r=\" + r);\n    }\n    if (r.getMetadataLength() == 0) {\n      throw new IOException(\"Metafile is empty, r=\" + r);\n    }\n  }\n\n  /**\n   * We're informed that a block is no longer valid. Delete it.\n   */\n  @Override // FsDatasetSpi\n  public void invalidate(String bpid, Block invalidBlks[]) throws IOException {\n    invalidate(bpid, invalidBlks, true);\n  }\n\n  private void invalidate(String bpid, Block[] invalidBlks, boolean async)\n      throws IOException {\n    final List<String> errors = new ArrayList<String>();\n    for (int i = 0; i < invalidBlks.length; i++) {","sourceCodeStart":2321,"sourceCodeEnd":2357,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L2321-L2357","documentation":"IOException thrown by FsDatasetImpl.checkReplicaFiles when the replica's metadata (checksum) file is missing: metadataExists() returns false, and the message embeds r.getMetadataURI(). Without the .meta checksum file the block cannot be validated or served for reads that verify checksums, so recovery/finalize paths refuse to proceed.","triggerScenarios":"checkReplicaFiles(r) during initReplicaRecovery/finalize on a replica whose block file exists but whose .meta file (blockId_generationStamp.meta) was deleted or never written - e.g., crash between data file creation and meta file creation.","commonSituations":"DataNode crash mid-write before the meta file was flushed; external deletion of *.meta; restore-from-backup that missed meta files; filesystem corruption after power loss.","solutions":["Check whether the .meta file path in the message exists elsewhere on the volume (moved by a broken tool) and restore it.","Run hdfs fsck -blocks: if other healthy replicas exist, invalidate this one and let re-replication rebuild it.","If this was the last replica, hdfs debug recoverLease / restart may allow reading data without checksums as a salvage path.","Fix the root cause (crash-consistency, external interference) and restart the DataNode."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":"boolean isMissingMetaFile(IOException e) {\n  return e.getMessage() != null && e.getMessage().endsWith(\".meta does not exist\") ||\n      (e.getMessage() != null && e.getMessage().contains(\"does not exist, r=\"));\n}","tryCatchPattern":"try {\n  dataset.initReplicaRecovery(rBlock);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"does not exist, r=\")) {\n    // missing .meta: exclude this replica from recovery, fsck + re-replicate\n    return recoverFromOtherReplicas(rBlock);\n  }\n  throw e;\n}","preventionTips":["Include *.meta files in any volume-level backup/restore procedure (or exclude the whole data dir).","Ensure meta and data files share the same filesystem so they fail together, not independently.","Run periodic fsck with -blockcheck to catch header/meta anomalies."],"tags":["hdfs","datanode","replica-files","checksum-meta","file-not-found"],"backgroundTag":"missing-checksum-metadata","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}