apache/hadoop · error · ReplicaAlreadyExistsException

Block {b} already exists in state {state} and thus cannot be

Error message

Block {b} already exists in state {state} and thus cannot be created.

What it means

When a client asks a DataNode to create a brand-new RBW replica, an existing volumeMap entry for the same blockPoolId+blockId is only acceptable if the request is a retry carrying a newer generation stamp (newGS != 0, which triggers cleanupReplica of the old copy). With newGS == 0 - a first-time create - any pre-existing replica is a contradiction, and ReplicaAlreadyExistsException is thrown.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:1633

  }

  @Override // FsDatasetSpi
  public ReplicaHandler createRbw(
      StorageType storageType, String storageId, ExtendedBlock b,
      boolean allowLazyPersist, long newGS) throws IOException {
    long startTimeMs = Time.monotonicNow();
    try (AutoCloseableLock lock = lockManager.readLock(LockLevel.BLOCK_POOl,
        b.getBlockPoolId())) {
      ReplicaInfo replicaInfo = volumeMap.get(b.getBlockPoolId(),
          b.getBlockId());
      if (replicaInfo != null) {
        // In case of retries with same blockPoolId + blockId as before
        // with updated GS, cleanup the old replica to avoid
        // any multiple copies with same blockPoolId + blockId
        if (newGS != 0L) {
          cleanupReplica(b.getBlockPoolId(), replicaInfo);
        } else {
          throw new ReplicaAlreadyExistsException("Block " + b +
              " already exists in state " + replicaInfo.getState() +
              " and thus cannot be created.");
        }
      }
      // create a new block
      FsVolumeReference ref = null;

      // Use ramdisk only if block size is a multiple of OS page size.
      // This simplifies reservation for partially used replicas
      // significantly.
      if (allowLazyPersist &&
          lazyWriter != null &&
          b.getNumBytes() % cacheManager.getOsPageSize() == 0 &&
          reserveLockedMemory(b.getNumBytes())) {
        try {
          // First try to place the block on a transient volume.
          ref = volumes.getNextTransientVolume(b.getNumBytes());
          datanode.getMetrics().incrRamDiskBlocksWrite();

View on GitHub (pinned to 2add963021)

Solutions

Retry the create with a bumped GS - with newGS != 0 the DataNode cleans up the old replica itself (this is the designed path; DFSOutputStream requests a new GS when it catches this exception)
If retries loop, inspect the DN's rbw/finalized directories for the blockId and remove the stray replica after fsck confirms no valid copy needs it
Run 'hdfs fsck -files -blocks -locations' to make sure the blockId is not double-allocated
Check DN logs for the earlier create attempt that left the replica behind

Example fix

// before: retrying createRbw with newGS=0 forever
fsDataset.createRbw(storageType, storageId, b, allowLazyPersist, 0L);
// -> ReplicaAlreadyExistsException: Block ... already exists ...

// after: on 'already exists', bump the GS so the DN cleans the old replica
try {
  fsDataset.createRbw(storageType, storageId, b, allowLazyPersist, 0L);
} catch (ReplicaAlreadyExistsException e) {
  long bumpedGs = b.getGenerationStamp() + 1;
  ExtendedBlock nb = new ExtendedBlock(
      b.getBlockPoolId(), b.getBlockId(), b.getNumBytes(), bumpedGs);
  fsDataset.createRbw(storageType, storageId, nb, allowLazyPersist, bumpedGs);
}

Defensive patterns

Strategy: retry

Validate before calling

Replica existing = fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());
if (existing != null && newGS == 0L) {
  // a first-time create cannot replace an existing replica: caller must bump the GS
  throw new IllegalStateException(
      "replica already exists for " + b + "; retry createRbw with a non-zero newGS");
}
fsDataset.createRbw(storageType, storageId, b, allowLazyPersist, newGS);

Try / catch

catch (ReplicaAlreadyExistsException e) {
  if (e.getMessage() != null && e.getMessage().contains("already exists")) {
    long bumpedGs = b.getGenerationStamp() + 1;
    retryCreateRbwWithNewGS(bumpedGs); // newGS != 0 makes the DN clean the old replica
  } else { throw e; }
}

Prevention

Make block creates idempotent: carry a fresh GS on every retry so stale replicas are cleaned, not collided with
Do not cache and replay OP_WRITE_BLOCK pipeline setups verbatim after failure
Clean stray rbw files in DN storage directories when rebuilding or reusing a DataNode's data

When it happens

Trigger: BlockReceiver.java:221 createRbw(storageType, storageId, b, allowLazyPersist, newGS) for a fresh block whose id already has a replica in this DN's volumeMap: an earlier abandoned create left a replica behind, and the retry arrived without a bumped GS.

Common situations: Pipeline setup failed on another DN and the retry re-lands on this DN before the client bumps the GS; stale volumeMap entries after DN restart with leftover rbw files; very rarely, duplicate block allocation by the NN.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5556054c2fb2e4e4. Report an issue: GitHub.