{"record":{"id":"7467f029a36fde30","repo":"apache/hadoop","slug":"replica-b-is-not-in-expected-state-state","errorCode":null,"errorMessage":"Replica {b} is not in expected state {state}","messagePattern":"Replica (.+?) is not in expected state (.+?)","errorType":"exception","errorClass":"UnexpectedReplicaStateException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":2260,"sourceCode":"   * @throws UnexpectedReplicaStateException   If the replica is not in the \n   *                                             expected state.\n   * @throws FileNotFoundException             If the block file is not found or there\n   *                                              was an error locating it.\n   * @throws EOFException                      If the replica length is too short.\n   * \n   * @throws IOException                       May be thrown from the methods called. \n   */\n  @Override // FsDatasetSpi\n  public void checkBlock(ExtendedBlock b, long minLength, ReplicaState state)\n      throws ReplicaNotFoundException, UnexpectedReplicaStateException,\n      FileNotFoundException, EOFException, IOException {\n    final ReplicaInfo replicaInfo = volumeMap.get(b.getBlockPoolId(), \n        b.getLocalBlock());\n    if (replicaInfo == null) {\n      throw new ReplicaNotFoundException(b);\n    }\n    if (replicaInfo.getState() != state) {\n      throw new UnexpectedReplicaStateException(b,state);\n    }\n    if (!replicaInfo.blockDataExists()) {\n      throw new FileNotFoundException(replicaInfo.getBlockURI().toString());\n    }\n    long onDiskLength = getLength(b);\n    if (onDiskLength < minLength) {\n      throw new EOFException(b + \"'s on-disk length \" + onDiskLength\n          + \" is shorter than minLength \" + minLength);\n    }\n  }\n\n  /**\n   * Check whether the given block is a valid one.\n   * valid means finalized\n   */\n  @Override // FsDatasetSpi\n  public boolean isValidBlock(ExtendedBlock b) {\n    // If block passed is null, we should return false.","sourceCodeStart":2242,"sourceCodeEnd":2278,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L2242-L2278","documentation":"UnexpectedReplicaStateException thrown by FsDatasetImpl.checkBlock when the replica exists in the volumeMap but its state differs from the state the caller requires (the 'state' parameter, typically FINALIZED for read paths). It means the block was found but is not in the lifecycle state the operation expects.","triggerScenarios":"checkBlock(b, minLength, ReplicaState.FINALIZED) while the replica is RBW (still being written), TEMPORARY (transfer in progress), or RUR (under recovery) - e.g., a read request hitting a block whose lease recovery has not finished.","commonSituations":"Reading a file while its writer still holds the lease (or crashed, and lease recovery is pending); block transfer and read racing; hflush-derived reads of unfinalized blocks via a path that requires FINALIZED.","solutions":["Wait for the writer to close the file or for lease recovery to complete (soft limit ~ minutes, enforceable via recoverLease).","As a client, call recoverLease on the file path to force recovery, then retry.","Verify the reader uses readShortCircuitStreams/plain read of visible length rather than assuming finalized state for appends.","If the replica is stuck RUR for a long time, restart the DataNode to clear the recovery state."],"exampleFix":"// before\ntry { dataset.checkBlock(b, len, ReplicaState.FINALIZED); }\ncatch (UnexpectedReplicaStateException e) { /* generic */ }\n\n// after - retry after forcing lease recovery\ntry { dataset.checkBlock(b, len, ReplicaState.FINALIZED); }\ncatch (UnexpectedReplicaStateException e) {\n  dfs.recoverLease(filePath); // unblocks RBW/RUR -> FINALIZED\n  // then retry once recovery completes\n}","handlingStrategy":"try-catch","validationCode":"// Where you own the state expectation, verify state first:\nReplicaInfo r = dataset.getReplicaInfo(b);\nif (r != null && r.getState() != ReplicaState.FINALIZED) {\n  // not finalizable yet: wait for lease recovery instead of triggering the exception\n}","typeGuard":"boolean isUnexpectedReplicaState(IOException e) {\n  return e instanceof org.apache.hadoop.hdfs.server.datanode.UnexpectedReplicaStateException;\n}","tryCatchPattern":"try {\n  dataset.checkBlock(b, minLength, ReplicaState.FINALIZED);\n} catch (UnexpectedReplicaStateException e) {\n  // RBW/RUR/TEMPORARY: force lease recovery, then retry once finalized\n  dfs.recoverLease(filePath);\n  retryAfterRecovery(b, minLength);\n}","preventionTips":["Do not read files whose writer is still active unless you handle unfinalized states.","Use recoverLease to unblock files abandoned by crashed writers.","Test read paths against blocks mid-recovery to ensure they rotate to a finalized replica."],"tags":["hdfs","datanode","block-check","replica-state","lease-recovery"],"backgroundTag":"replica-state-machine-violation","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}