apache/hadoop · error · UnexpectedReplicaStateException

Replica {b} is not in expected state {state}

Error message

Replica {b} is not in expected state {state}

What it means

UnexpectedReplicaStateException thrown by FsDatasetImpl.checkBlock when the replica exists in the volumeMap but its state differs from the state the caller requires (the 'state' parameter, typically FINALIZED for read paths). It means the block was found but is not in the lifecycle state the operation expects.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:2260

   * @throws UnexpectedReplicaStateException   If the replica is not in the 
   *                                             expected state.
   * @throws FileNotFoundException             If the block file is not found or there
   *                                              was an error locating it.
   * @throws EOFException                      If the replica length is too short.
   * 
   * @throws IOException                       May be thrown from the methods called. 
   */
  @Override // FsDatasetSpi
  public void checkBlock(ExtendedBlock b, long minLength, ReplicaState state)
      throws ReplicaNotFoundException, UnexpectedReplicaStateException,
      FileNotFoundException, EOFException, IOException {
    final ReplicaInfo replicaInfo = volumeMap.get(b.getBlockPoolId(), 
        b.getLocalBlock());
    if (replicaInfo == null) {
      throw new ReplicaNotFoundException(b);
    }
    if (replicaInfo.getState() != state) {
      throw new UnexpectedReplicaStateException(b,state);
    }
    if (!replicaInfo.blockDataExists()) {
      throw new FileNotFoundException(replicaInfo.getBlockURI().toString());
    }
    long onDiskLength = getLength(b);
    if (onDiskLength < minLength) {
      throw new EOFException(b + "'s on-disk length " + onDiskLength
          + " is shorter than minLength " + minLength);
    }
  }

  /**
   * Check whether the given block is a valid one.
   * valid means finalized
   */
  @Override // FsDatasetSpi
  public boolean isValidBlock(ExtendedBlock b) {
    // If block passed is null, we should return false.

View on GitHub (pinned to 2add963021)

Solutions

1. Wait for the writer to close the file or for lease recovery to complete (soft limit ~ minutes, enforceable via recoverLease).
2. As a client, call recoverLease on the file path to force recovery, then retry.
3. Verify the reader uses readShortCircuitStreams/plain read of visible length rather than assuming finalized state for appends.
4. If the replica is stuck RUR for a long time, restart the DataNode to clear the recovery state.

Example fix

// before
try { dataset.checkBlock(b, len, ReplicaState.FINALIZED); }
catch (UnexpectedReplicaStateException e) { /* generic */ }

// after - retry after forcing lease recovery
try { dataset.checkBlock(b, len, ReplicaState.FINALIZED); }
catch (UnexpectedReplicaStateException e) {
  dfs.recoverLease(filePath); // unblocks RBW/RUR -> FINALIZED
  // then retry once recovery completes
}

// Where you own the state expectation, verify state first:
ReplicaInfo r = dataset.getReplicaInfo(b);
if (r != null && r.getState() != ReplicaState.FINALIZED) {
  // not finalizable yet: wait for lease recovery instead of triggering the exception
}

boolean isUnexpectedReplicaState(IOException e) {
  return e instanceof org.apache.hadoop.hdfs.server.datanode.UnexpectedReplicaStateException;
}

try {
  dataset.checkBlock(b, minLength, ReplicaState.FINALIZED);
} catch (UnexpectedReplicaStateException e) {
  // RBW/RUR/TEMPORARY: force lease recovery, then retry once finalized
  dfs.recoverLease(filePath);
  retryAfterRecovery(b, minLength);
}

When it happens

Trigger: checkBlock(b, minLength, ReplicaState.FINALIZED) while the replica is RBW (still being written), TEMPORARY (transfer in progress), or RUR (under recovery) - e.g., a read request hitting a block whose lease recovery has not finished.

Common situations: Reading a file while its writer still holds the lease (or crashed, and lease recovery is pending); block transfer and read racing; hflush-derived reads of unfinalized blocks via a path that requires FINALIZED.

Related errors

• Cannot append to an unfinalized replica {b}
• replica.getState() != RUR, replica={replica}
• Replica was found but missing fields.
• Replica is not readable, block={block}, replica={replica}
• Failed to create temporary file for {}. File {} should be c

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7467f029a36fde30. Report an issue: GitHub.