{"record":{"id":"d64ca8b18f25caeb","repo":"apache/hadoop","slug":"replica-was-found-but-missing-fields","errorCode":null,"errorMessage":"Replica was found but missing fields. ","messagePattern":"Replica was found but missing fields\\. ","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/InterDatanodeProtocolTranslatorPB.java","lineNumber":88,"sourceCode":"\n  @Override\n  public void close() {\n    RPC.stopProxy(rpcProxy);\n  }\n\n  @Override\n  public ReplicaRecoveryInfo initReplicaRecovery(RecoveringBlock rBlock)\n      throws IOException {\n    InitReplicaRecoveryRequestProto req = InitReplicaRecoveryRequestProto\n        .newBuilder().setBlock(PBHelper.convert(rBlock)).build();\n    InitReplicaRecoveryResponseProto resp;\n    resp = ipc(() -> rpcProxy.initReplicaRecovery(NULL_CONTROLLER, req));\n    if (!resp.getReplicaFound()) {\n      // No replica found on the remote node.\n      return null;\n    } else {\n      if (!resp.hasBlock() || !resp.hasState()) {\n        throw new IOException(\"Replica was found but missing fields. \" +\n            \"Req: \" + req + \"\\n\" +\n            \"Resp: \" + resp);\n      }\n    }\n    \n    BlockProto b = resp.getBlock();\n    return new ReplicaRecoveryInfo(b.getBlockId(), b.getNumBytes(),\n        b.getGenStamp(), PBHelper.convert(resp.getState()));\n  }\n\n  @Override\n  public String updateReplicaUnderRecovery(ExtendedBlock oldBlock,\n      long recoveryId, long newBlockId, long newLength) throws IOException {\n    UpdateReplicaUnderRecoveryRequestProto req = \n        UpdateReplicaUnderRecoveryRequestProto.newBuilder()\n        .setBlock(PBHelperClient.convert(oldBlock))\n        .setNewLength(newLength).setNewBlockId(newBlockId)\n        .setRecoveryId(recoveryId).build();","sourceCodeStart":70,"sourceCodeEnd":106,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/InterDatanodeProtocolTranslatorPB.java#L70-L106","documentation":"Client translator for InterDatanodeProtocol, used during block/lease recovery. initReplicaRecovery() asks the remote datanode for replica info; the response must either report replicaFound=false or carry both the block fields (id, bytes, genStamp) and the replica state. A response that claims a replica but omits block or state violates the protobuf contract, so recovery aborts with IOException dumping the full Req and Resp.","triggerScenarios":"The datanode's InitReplicaRecoveryResponseProto sets replicaFound=true but has no block or no state field: mixed hadoop-hdfs versions on the two datanodes, or a buggy/forked datanode build that hand-crafts the response.","commonSituations":"Rolling upgrades where one datanode in the pipeline is older/newer; lease recovery or pipeline recovery immediately after upgrading a single DN; third-party datanode forks.","solutions":["Compare releases on every datanode in the pipeline (hdfs version) and complete or roll back the rolling upgrade so versions match.","Read the Req/Resp dump in the exception message, then check the responding datanode's logs at that timestamp.","Restart the datanode that produced the malformed response.","If it persists, capture the RPC payload and compare against InitReplicaRecoveryResponseProto for the installed release."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  ReplicaRecoveryInfo info = datanode.initReplicaRecovery(rBlock);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Replica was found but missing fields\")) {\n    // protocol-level inconsistency: do not retry against this datanode; log Req/Resp and abort recovery for this replica\n    LOG.error(\"Malformed initReplicaRecovery response from \" + datanode, e);\n    throw e;\n  }\n  throw e;\n}","preventionTips":["Keep every datanode in a recovery pipeline on the same Hadoop release.","Treat this message as a version-skew signal: check 'hdfs version' on both DN hosts immediately.","Never hand-build InitReplicaRecoveryResponseProto in forks; always use the generated builders with all fields set."],"tags":["hdfs","block-recovery","lease-recovery","datanode","protobuf","rpc"],"backgroundTag":"rpc-protocol-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}