apache/hadoop · error · IOException

CreateRBW returned a replica of state {state} for block {blo

Error message

CreateRBW returned a replica of state {state} for block {blockId}

What it means

Defensive invariant of createRbw: the freshly created replica returned by FsVolumeImpl.createRbw() must be in RBW state before being added to the volumeMap. Any other state means the volume implementation violated the write-pipeline contract, so the DataNode aborts the create instead of registering a bogus replica.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:1682

      if (ref == null) {
        ref = volumes.getNextVolume(storageType, storageId, b.getNumBytes());
      }
      LOG.debug("Creating Rbw, block: {} on volume: {}", b, ref.getVolume());

      FsVolumeImpl v = (FsVolumeImpl) ref.getVolume();
      // create an rbw file to hold block in the designated volume

      if (allowLazyPersist && !v.isTransientStorage()) {
        datanode.getMetrics().incrRamDiskBlocksWriteFallback();
      }

      ReplicaInPipeline newReplicaInfo;
      try (AutoCloseableLock l = lockManager.writeLock(LockLevel.DIR,
          b.getBlockPoolId(), v.getStorageID(),
          datasetSubLockStrategy.blockIdToSubLock(b.getBlockId()))) {
        newReplicaInfo = v.createRbw(b);
        if (newReplicaInfo.getReplicaInfo().getState() != ReplicaState.RBW) {
          throw new IOException("CreateRBW returned a replica of state "
              + newReplicaInfo.getReplicaInfo().getState()
              + " for block " + b.getBlockId());
        }
        volumeMap.add(b.getBlockPoolId(), newReplicaInfo.getReplicaInfo());
        return new ReplicaHandler(newReplicaInfo, ref);
      } catch (IOException e) {
        IOUtils.cleanupWithLogger(null, ref);
        throw e;
      }
    } finally {
      if (dataNodeMetrics != null) {
        long createRbwMs = Time.monotonicNow() - startTimeMs;
        dataNodeMetrics.addCreateRbwOp(createRbwMs);
      }
    }
  }

  @Override // FsDatasetSpi

View on GitHub (pinned to 2add963021)

Solutions

1. Reproduce with the stock FsDatasetFactory - if it passes, the bug is in the custom dataset
2. Fix the custom volume's createRbw to construct and return an RBW-state replica
3. Collect DataNode logs and file a bug with the custom dataset's maintainer
4. On stock builds, upgrade and open an HDFS JIRA with the log line

catch (IOException ioe) {
  if (ioe.getMessage() != null && ioe.getMessage().startsWith("CreateRBW returned")) {
    LOG.error("FsVolume.createRbw returned non-RBW replica; dataset bug", ioe);
    failOverToOtherReplicas(); // do not blind-retry an invariant violation
  } else { throw ioe; }
}

When it happens

Trigger: v.createRbw(b) returning a non-RBW ReplicaInPipeline - only plausible with a custom FsDataset/FsVolume (dfs.datanode.fsdataset.factory) or a genuine HDFS bug; stock FileFsVolume always returns RBW.

Common situations: Third-party FsDataset implementations (in-memory backends, tiered storage integrations, test doubles) returning wrong states; never seen with the stock FileSystemDataset volumes.

Related errors

• Append on block {blockId} returned a replica of state {state
• r.getVolume() = null, temp={temp}
• Failed to create temporary file for {}. File {} should not
• Failed to create temporary file for {}. File {} should be c
• Block {b} already exists in state {state} and thus cannot be

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a6ad83a985507716. Report an issue: GitHub.