{"record":{"id":"a6ad83a985507716","repo":"apache/hadoop","slug":"createrbw-returned-a-replica-of-state-state-for","errorCode":null,"errorMessage":"CreateRBW returned a replica of state {state} for block {blockId}","messagePattern":"CreateRBW returned a replica of state (.+?) for block (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":1682,"sourceCode":"      if (ref == null) {\n        ref = volumes.getNextVolume(storageType, storageId, b.getNumBytes());\n      }\n      LOG.debug(\"Creating Rbw, block: {} on volume: {}\", b, ref.getVolume());\n\n      FsVolumeImpl v = (FsVolumeImpl) ref.getVolume();\n      // create an rbw file to hold block in the designated volume\n\n      if (allowLazyPersist && !v.isTransientStorage()) {\n        datanode.getMetrics().incrRamDiskBlocksWriteFallback();\n      }\n\n      ReplicaInPipeline newReplicaInfo;\n      try (AutoCloseableLock l = lockManager.writeLock(LockLevel.DIR,\n          b.getBlockPoolId(), v.getStorageID(),\n          datasetSubLockStrategy.blockIdToSubLock(b.getBlockId()))) {\n        newReplicaInfo = v.createRbw(b);\n        if (newReplicaInfo.getReplicaInfo().getState() != ReplicaState.RBW) {\n          throw new IOException(\"CreateRBW returned a replica of state \"\n              + newReplicaInfo.getReplicaInfo().getState()\n              + \" for block \" + b.getBlockId());\n        }\n        volumeMap.add(b.getBlockPoolId(), newReplicaInfo.getReplicaInfo());\n        return new ReplicaHandler(newReplicaInfo, ref);\n      } catch (IOException e) {\n        IOUtils.cleanupWithLogger(null, ref);\n        throw e;\n      }\n    } finally {\n      if (dataNodeMetrics != null) {\n        long createRbwMs = Time.monotonicNow() - startTimeMs;\n        dataNodeMetrics.addCreateRbwOp(createRbwMs);\n      }\n    }\n  }\n\n  @Override // FsDatasetSpi","sourceCodeStart":1664,"sourceCodeEnd":1700,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L1664-L1700","documentation":"Defensive invariant of createRbw: the freshly created replica returned by FsVolumeImpl.createRbw() must be in RBW state before being added to the volumeMap. Any other state means the volume implementation violated the write-pipeline contract, so the DataNode aborts the create instead of registering a bogus replica.","triggerScenarios":"v.createRbw(b) returning a non-RBW ReplicaInPipeline - only plausible with a custom FsDataset/FsVolume (dfs.datanode.fsdataset.factory) or a genuine HDFS bug; stock FileFsVolume always returns RBW.","commonSituations":"Third-party FsDataset implementations (in-memory backends, tiered storage integrations, test doubles) returning wrong states; never seen with the stock FileSystemDataset volumes.","solutions":["Reproduce with the stock FsDatasetFactory - if it passes, the bug is in the custom dataset","Fix the custom volume's createRbw to construct and return an RBW-state replica","Collect DataNode logs and file a bug with the custom dataset's maintainer","On stock builds, upgrade and open an HDFS JIRA with the log line"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (IOException ioe) {\n  if (ioe.getMessage() != null && ioe.getMessage().startsWith(\"CreateRBW returned\")) {\n    LOG.error(\"FsVolume.createRbw returned non-RBW replica; dataset bug\", ioe);\n    failOverToOtherReplicas(); // do not blind-retry an invariant violation\n  } else { throw ioe; }\n}","preventionTips":["Unit-test custom FsVolume.createRbw to return RBW state before deploying","Alert on 'CreateRBW returned' invariant messages - they mark dataset bugs, not transient faults","Run integration tests against the custom dataset with concurrent create/recovery workloads"],"tags":["hdfs","datanode","block-write","invariant","custom-fsdataset"],"backgroundTag":"hdfs-replica-state-invariant","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}