apache/hadoop · error · ReplicaNotFoundException

Replica does not exist {}

Error message

Replica does not exist {}

What it means

Thrown as ReplicaNotFoundException (NON_EXISTENT_REPLICA prefix) from FsDatasetImpl.getReplicaInfo(ExtendedBlock) when volumeMap.get(bpid, localBlock) returns null and a second lookup by blockId alone also returns null — proving the block ID itself is unknown, not just the generation stamp. This distinguishes a genuinely absent replica from the UNEXPECTED_GS case (error 2406).

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:894

    }
    return info.getDataInputStream(seekOffset);
  }

  /**
   * Get the meta info of a block stored in volumeMap. To find a block,
   * block pool Id, block Id and generation stamp must match.
   * @param b extended block
   * @return the meta replica information
   * @throws ReplicaNotFoundException if no entry is in the map or 
   *                        there is a generation stamp mismatch
   */
  ReplicaInfo getReplicaInfo(ExtendedBlock b)
      throws ReplicaNotFoundException {
    ReplicaInfo info = volumeMap.get(b.getBlockPoolId(), b.getLocalBlock());
    if (info == null) {
      if (volumeMap.get(b.getBlockPoolId(), b.getLocalBlock().getBlockId())
          == null) {
        throw new ReplicaNotFoundException(
            ReplicaNotFoundException.NON_EXISTENT_REPLICA + b);
      } else {
        throw new ReplicaNotFoundException(
            ReplicaNotFoundException.UNEXPECTED_GS_REPLICA + b);
      }
    }
    return info;
  }

  /**
   * Get the meta info of a block stored in volumeMap. Block is looked up
   * without matching the generation stamp.
   * @param bpid block pool Id
   * @param blkid block Id
   * @return the meta replica information; null if block was not found
   * @throws ReplicaNotFoundException if no entry is in the map or 
   *                        there is a generation stamp mismatch
   */

View on GitHub (pinned to 2add963021)

Solutions

1. Check the DataNode log for 'Deleting block' / invalidation messages around the block ID to confirm intentional removal.
2. Force a block report from the DataNode (hdfs dfsadmin -triggerBlockReport) so NameNode state matches reality, then retry the client operation.
3. For appends: recover the lease (hdfs debug recoverLease -path <file>) so the NameNode re-establishes a valid pipeline instead of hammering the stale replica.
4. If replicas are genuinely missing, run hdfs fsck -blocks and let HDFS re-replicate from healthy replicas.

// Guard append/recover-style calls: confirm the replica exists first.
ReplicaInfo info = fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());
if (info == null) {
  // block gone on this DN: get fresh targets from the NameNode
  return reestablishPipeline(dfsClient, b);
}

// ReplicaNotFoundException is the specific signal; recover at the NN level.
try {
  return fsDataset.getReplicaInfo(b);
} catch (ReplicaNotFoundException e) {
  // NON_EXISTENT vs UNEXPECTED_GS differ by message prefix in the cause
  LOG.warn("Replica missing on DN, recovering pipeline: {}", e.getMessage());
  return recoverViaNameNode(dfsClient, b);
}

When it happens

Trigger: Any internal FsDatasetImpl path that resolves a replica before operating: append, recoverBlock, finalizeBlock, moveBlockAcrossStorage, getMetaDataInputStream, etc., invoked for a block ID with no volumeMap entry in that block pool.

Common situations: Client retries/append after the replica was deleted; DataNode failed volumes removed replicas from the map; block invalidated (datanode 'deleting block' log lines) while a writer still holds it; wrong-blockpool ExtendedBlock after cluster re-format.

Related errors

• Replica does not exist {}:{}
• BlockId {} is not valid.
• No data exists for block {}
• Found duplicated storage UUID: %s in %s.
• Storage type %s already exists on same mount: %s.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/39dc18cc2d8fb801. Report an issue: GitHub.