apache/hadoop · error · IOException

Expected replica state: RBW obtained {state} for converting

Error message

Expected replica state: RBW obtained {state} for converting block {b}

What it means

Thrown by FsDatasetImpl.convertTemporaryToRbw when the DataNode converts a TEMPORARY replica (written by a block transfer) into an RBW replica for append, but the object returned by FsVolumeImpl.convertTemporaryToRbw does not report ReplicaState.RBW. This is an internal state-machine invariant check: the conversion itself completed, but the resulting replica is in an unexpected state. It indicates corrupted in-memory replica bookkeeping (volumeMap) rather than a bad client request.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:1855

      // set writer to the current thread
      // temp.setWriter(Thread.currentThread());

      // check length
      final long numBytes = temp.getNumBytes();
      if (numBytes < visible) {
        throw new IOException(numBytes + " = numBytes < visible = "
            + visible + ", temp=" + temp);
      }
      // check volume
      final FsVolumeImpl v = (FsVolumeImpl) temp.getVolume();
      if (v == null) {
        throw new IOException("r.getVolume() = null, temp=" + temp);
      }

      final ReplicaInPipeline rbw = v.convertTemporaryToRbw(b, temp);

      if(rbw.getState() != ReplicaState.RBW) {
        throw new IOException("Expected replica state: " + ReplicaState.RBW
            + " obtained " + rbw.getState() + " for converting block "
            + b);
      }
      // overwrite the RBW in the volume map
      volumeMap.add(b.getBlockPoolId(), rbw.getReplicaInfo());
      return rbw;
    } finally {
      if (dataNodeMetrics != null) {
        long convertTemporaryToRbwMs = Time.monotonicNow() - startTimeMs;
        dataNodeMetrics.addConvertTemporaryToRbwOp(convertTemporaryToRbwMs);
      }
    }
  }

  private boolean isReplicaProvided(ReplicaInfo replicaInfo) {
    if (replicaInfo == null) {
      return false;
    }

View on GitHub (pinned to 2add963021)

Solutions

Restart the DataNode so the volume map is rebuilt from disk state (FsVolumeImpl block scanning re-derives replica states).
Inspect the DataNode log for the preceding lines (numBytes/visible checks, 'r.getVolume() = null') to see which earlier invariant was near-violation.
Check whether the block file on disk is in a weird state (e.g., RBW and TEMPORARY files for the same block id) and let the NameNode re-replicate, then delete the orphaned replica.
If reproducible on stock hardware, capture the full stack trace and file a HDFS JIRA - this is a bookkeeping bug, not an operational config issue.

Defensive patterns

Strategy: try-catch

Try / catch

try {
  ReplicaInPipeline rbw = dataset.convertTemporaryToRbw(b, temp);
} catch (IOException e) {
  // invariant break: state != RBW after conversion. Restart-grade failure.
  LOG.error("Replica conversion invariant broken for " + b, e);
  throw e; // do not silently continue; recovery must pick another replica
}

Prevention

Keep DataNodes on a crash-consistent filesystem so transfer state never half-converts.
Avoid killing DataNodes mid block transfer; use graceful stop (hdfs --daemon stop datanode).
Monitor for this message and restart the DataNode promptly when it appears.

When it happens

Trigger: A client calls recoverClose/append on a block whose replica reached the DataNode via pipeline transfer (TEMPORARY state); lease recovery triggers convertTemporaryToRbw; on return, rbw.getState() is anything other than RBW. Only reachable through the internal FsDatasetSpi.convertTemporaryToRbw path driven by BlockRecoveryWorker and DataXceiver's OP_WRITE_BLOCK with TRANSFER_FINALIZED=false.

Common situations: Appears after a DataNode crash/restart mid-transfer left a half-converted replica in the volume map; after disk-error handling detached a volume while a transfer was finishing; or after downgrade/upgrade between HDFS versions where replica metadata was reloaded inconsistently.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5dda9c4ac4ebcec5. Report an issue: GitHub.