apache/hadoop · error · ReplicaAlreadyExistsException

RBW replica {replicaInfo}bytesRcvd({numBytes}), bytesOnDisk(

Error message

RBW replica {replicaInfo}bytesRcvd({numBytes}), bytesOnDisk({bytesOnDisk}), and bytesAcked({bytesAcked}) are not the same.

What it means

Before recovering an RBW replica, recoverCheck requires its three length counters to agree: bytesRcvd (getNumBytes), bytesOnDisk and bytesAcked must all equal replicaLen. If they differ, the replica holds an unacknowledged or unflushed tail and cannot be recovered as-is, so ReplicaAlreadyExistsException is thrown.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:1531

    if (replicaGenerationStamp < b.getGenerationStamp() ||
        replicaGenerationStamp > newGS) {
      throw new ReplicaNotFoundException(
          ReplicaNotFoundException.UNEXPECTED_GS_REPLICA + replicaGenerationStamp
          + ". Expected GS range is [" + b.getGenerationStamp() + ", " + 
          newGS + "].");
    }
    
    // stop the previous writer before check a replica's length
    long replicaLen = replicaInfo.getNumBytes();
    if (replicaInfo.getState() == ReplicaState.RBW) {
      ReplicaInPipeline rbw = (ReplicaInPipeline) replicaInfo;
      if (!rbw.attemptToSetWriter(null, Thread.currentThread())) {
        throw new MustStopExistingWriter(rbw);
      }
      // check length: bytesRcvd, bytesOnDisk, and bytesAcked should be the same
      if (replicaLen != rbw.getBytesOnDisk() 
          || replicaLen != rbw.getBytesAcked()) {
        throw new ReplicaAlreadyExistsException("RBW replica " + replicaInfo + 
            "bytesRcvd(" + rbw.getNumBytes() + "), bytesOnDisk(" + 
            rbw.getBytesOnDisk() + "), and bytesAcked(" + rbw.getBytesAcked() +
            ") are not the same.");
      }
    }
    
    // check block length
    if (replicaLen != expectedBlockLen) {
      throw new IOException("Corrupted replica " + replicaInfo + 
          " with a length of " + replicaLen + 
          " expected length is " + expectedBlockLen);
    }
    
    return replicaInfo;
  }

  @Override  // FsDatasetSpi
  public ReplicaHandler recoverAppend(

View on GitHub (pinned to 2add963021)

Solutions

Retry the recovery - once the writer is stopped and the tail truncated, the counters converge and the next attempt succeeds (the NN retries block recovery)
Verify the previous writer is really dead (no orphaned DataXceiver threads); take a DN thread dump if the error repeats
If permanently stuck, remove the RBW replica: the NN re-replicates from the committed length
Close/abort application streams deterministically so recovery never meets a half-written tail

Defensive patterns

Strategy: try-catch

Validate before calling

ReplicaInfo raw = (ReplicaInfo) fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());
if (raw != null && raw.getState() == ReplicaState.RBW) {
  ReplicaInPipeline rbw = (ReplicaInPipeline) raw;
  if (rbw.getNumBytes() != rbw.getBytesOnDisk()
      || rbw.getNumBytes() != rbw.getBytesAcked()) {
    stopWriterAndWaitForQuiesce(rbw); // counters converge, then recover
    return;
  }
}
fsDataset.recoverAppend(b, newGS, expectedBlockLen);

Type guard

boolean isQuiescentRbw(ReplicaInPipeline rbw) {
  return rbw.getState() == ReplicaState.RBW
      && rbw.getNumBytes() == rbw.getBytesOnDisk()
      && rbw.getNumBytes() == rbw.getBytesAcked();
}

Try / catch

catch (ReplicaAlreadyExistsException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("RBW replica")) {
    stopPreviousWriterThenRetryRecoveryOnce(); // tail truncation converges counters
  } else { throw e; }
}

Prevention

Abort or close client streams cleanly so RBW replicas are truncated/finalized deterministically
Before invoking recovery, ensure no writer thread still owns the replica (attemptToSetWriter semantics)
Retry recovery after stopping the previous writer instead of failing hard on first counter mismatch

When it happens

Trigger: recoverAppend/recoverClose touching an RBW replica where data was received but not flushed (numBytes > bytesOnDisk) or not yet acked upstream (numBytes > bytesAcked) - the typical aftermath of a writer crashing mid-packet, sampled at the wrong moment.

Common situations: Lease recovery racing a still-active writer; DataNode killed mid-packet; packet-ack lag making the counters transiently unequal exactly when recovery samples them.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3563e9b1a0280053. Report an issue: GitHub.