{"record":{"id":"5dda9c4ac4ebcec5","repo":"apache/hadoop","slug":"expected-replica-state-rbw-obtained-state-for-c","errorCode":null,"errorMessage":"Expected replica state: RBW obtained {state} for converting block {b}","messagePattern":"Expected replica state: RBW obtained (.+?) for converting block (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":1855,"sourceCode":"      // set writer to the current thread\n      // temp.setWriter(Thread.currentThread());\n\n      // check length\n      final long numBytes = temp.getNumBytes();\n      if (numBytes < visible) {\n        throw new IOException(numBytes + \" = numBytes < visible = \"\n            + visible + \", temp=\" + temp);\n      }\n      // check volume\n      final FsVolumeImpl v = (FsVolumeImpl) temp.getVolume();\n      if (v == null) {\n        throw new IOException(\"r.getVolume() = null, temp=\" + temp);\n      }\n\n      final ReplicaInPipeline rbw = v.convertTemporaryToRbw(b, temp);\n\n      if(rbw.getState() != ReplicaState.RBW) {\n        throw new IOException(\"Expected replica state: \" + ReplicaState.RBW\n            + \" obtained \" + rbw.getState() + \" for converting block \"\n            + b);\n      }\n      // overwrite the RBW in the volume map\n      volumeMap.add(b.getBlockPoolId(), rbw.getReplicaInfo());\n      return rbw;\n    } finally {\n      if (dataNodeMetrics != null) {\n        long convertTemporaryToRbwMs = Time.monotonicNow() - startTimeMs;\n        dataNodeMetrics.addConvertTemporaryToRbwOp(convertTemporaryToRbwMs);\n      }\n    }\n  }\n\n  private boolean isReplicaProvided(ReplicaInfo replicaInfo) {\n    if (replicaInfo == null) {\n      return false;\n    }","sourceCodeStart":1837,"sourceCodeEnd":1873,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L1837-L1873","documentation":"Thrown by FsDatasetImpl.convertTemporaryToRbw when the DataNode converts a TEMPORARY replica (written by a block transfer) into an RBW replica for append, but the object returned by FsVolumeImpl.convertTemporaryToRbw does not report ReplicaState.RBW. This is an internal state-machine invariant check: the conversion itself completed, but the resulting replica is in an unexpected state. It indicates corrupted in-memory replica bookkeeping (volumeMap) rather than a bad client request.","triggerScenarios":"A client calls recoverClose/append on a block whose replica reached the DataNode via pipeline transfer (TEMPORARY state); lease recovery triggers convertTemporaryToRbw; on return, rbw.getState() is anything other than RBW. Only reachable through the internal FsDatasetSpi.convertTemporaryToRbw path driven by BlockRecoveryWorker and DataXceiver's OP_WRITE_BLOCK with TRANSFER_FINALIZED=false.","commonSituations":"Appears after a DataNode crash/restart mid-transfer left a half-converted replica in the volume map; after disk-error handling detached a volume while a transfer was finishing; or after downgrade/upgrade between HDFS versions where replica metadata was reloaded inconsistently.","solutions":["Restart the DataNode so the volume map is rebuilt from disk state (FsVolumeImpl block scanning re-derives replica states).","Inspect the DataNode log for the preceding lines (numBytes/visible checks, 'r.getVolume() = null') to see which earlier invariant was near-violation.","Check whether the block file on disk is in a weird state (e.g., RBW and TEMPORARY files for the same block id) and let the NameNode re-replicate, then delete the orphaned replica.","If reproducible on stock hardware, capture the full stack trace and file a HDFS JIRA - this is a bookkeeping bug, not an operational config issue."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  ReplicaInPipeline rbw = dataset.convertTemporaryToRbw(b, temp);\n} catch (IOException e) {\n  // invariant break: state != RBW after conversion. Restart-grade failure.\n  LOG.error(\"Replica conversion invariant broken for \" + b, e);\n  throw e; // do not silently continue; recovery must pick another replica\n}","preventionTips":["Keep DataNodes on a crash-consistent filesystem so transfer state never half-converts.","Avoid killing DataNodes mid block transfer; use graceful stop (hdfs --daemon stop datanode).","Monitor for this message and restart the DataNode promptly when it appears."],"tags":["hdfs","datanode","block-recovery","replica-state","invariant"],"backgroundTag":"replica-state-machine-violation","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}