apache/hadoop · error · IOException

Replica was found but missing fields.

Error message

Replica was found but missing fields. 

What it means

Client translator for InterDatanodeProtocol, used during block/lease recovery. initReplicaRecovery() asks the remote datanode for replica info; the response must either report replicaFound=false or carry both the block fields (id, bytes, genStamp) and the replica state. A response that claims a replica but omits block or state violates the protobuf contract, so recovery aborts with IOException dumping the full Req and Resp.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/InterDatanodeProtocolTranslatorPB.java:88

  @Override
  public void close() {
    RPC.stopProxy(rpcProxy);
  }

  @Override
  public ReplicaRecoveryInfo initReplicaRecovery(RecoveringBlock rBlock)
      throws IOException {
    InitReplicaRecoveryRequestProto req = InitReplicaRecoveryRequestProto
        .newBuilder().setBlock(PBHelper.convert(rBlock)).build();
    InitReplicaRecoveryResponseProto resp;
    resp = ipc(() -> rpcProxy.initReplicaRecovery(NULL_CONTROLLER, req));
    if (!resp.getReplicaFound()) {
      // No replica found on the remote node.
      return null;
    } else {
      if (!resp.hasBlock() || !resp.hasState()) {
        throw new IOException("Replica was found but missing fields. " +
            "Req: " + req + "\n" +
            "Resp: " + resp);
      }
    }
    
    BlockProto b = resp.getBlock();
    return new ReplicaRecoveryInfo(b.getBlockId(), b.getNumBytes(),
        b.getGenStamp(), PBHelper.convert(resp.getState()));
  }

  @Override
  public String updateReplicaUnderRecovery(ExtendedBlock oldBlock,
      long recoveryId, long newBlockId, long newLength) throws IOException {
    UpdateReplicaUnderRecoveryRequestProto req = 
        UpdateReplicaUnderRecoveryRequestProto.newBuilder()
        .setBlock(PBHelperClient.convert(oldBlock))
        .setNewLength(newLength).setNewBlockId(newBlockId)
        .setRecoveryId(recoveryId).build();

View on GitHub (pinned to 2add963021)

Solutions

1. Compare releases on every datanode in the pipeline (hdfs version) and complete or roll back the rolling upgrade so versions match.
2. Read the Req/Resp dump in the exception message, then check the responding datanode's logs at that timestamp.
3. Restart the datanode that produced the malformed response.
4. If it persists, capture the RPC payload and compare against InitReplicaRecoveryResponseProto for the installed release.

try {
  ReplicaRecoveryInfo info = datanode.initReplicaRecovery(rBlock);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Replica was found but missing fields")) {
    // protocol-level inconsistency: do not retry against this datanode; log Req/Resp and abort recovery for this replica
    LOG.error("Malformed initReplicaRecovery response from " + datanode, e);
    throw e;
  }
  throw e;
}

When it happens

Trigger: The datanode's InitReplicaRecoveryResponseProto sets replicaFound=true but has no block or no state field: mixed hadoop-hdfs versions on the two datanodes, or a buggy/forked datanode build that hand-crafts the response.

Common situations: Rolling upgrades where one datanode in the pipeline is older/newer; lease recovery or pipeline recovery immediately after upgrading a single DN; third-party datanode forks.

Related errors

• Cannot append to a replica with unexpected generation stamp
• replica.getState() != RUR, replica={replica}
• rur.getRecoveryID() != recoveryId = {recoveryId}, rur={rur}
• {} has no enough internal blocks(current: {}), unable to sta
• Replica gen stamp < block genstamp, block={block}, replica={

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d64ca8b18f25caeb. Report an issue: GitHub.