{"record":{"id":"5556054c2fb2e4e4","repo":"apache/hadoop","slug":"block-b-already-exists-in-state-state-and-thus","errorCode":null,"errorMessage":"Block {b} already exists in state {state} and thus cannot be created.","messagePattern":"Block (.+?) already exists in state (.+?) and thus cannot be created\\.","errorType":"exception","errorClass":"ReplicaAlreadyExistsException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":1633,"sourceCode":"  }\n\n  @Override // FsDatasetSpi\n  public ReplicaHandler createRbw(\n      StorageType storageType, String storageId, ExtendedBlock b,\n      boolean allowLazyPersist, long newGS) throws IOException {\n    long startTimeMs = Time.monotonicNow();\n    try (AutoCloseableLock lock = lockManager.readLock(LockLevel.BLOCK_POOl,\n        b.getBlockPoolId())) {\n      ReplicaInfo replicaInfo = volumeMap.get(b.getBlockPoolId(),\n          b.getBlockId());\n      if (replicaInfo != null) {\n        // In case of retries with same blockPoolId + blockId as before\n        // with updated GS, cleanup the old replica to avoid\n        // any multiple copies with same blockPoolId + blockId\n        if (newGS != 0L) {\n          cleanupReplica(b.getBlockPoolId(), replicaInfo);\n        } else {\n          throw new ReplicaAlreadyExistsException(\"Block \" + b +\n              \" already exists in state \" + replicaInfo.getState() +\n              \" and thus cannot be created.\");\n        }\n      }\n      // create a new block\n      FsVolumeReference ref = null;\n\n      // Use ramdisk only if block size is a multiple of OS page size.\n      // This simplifies reservation for partially used replicas\n      // significantly.\n      if (allowLazyPersist &&\n          lazyWriter != null &&\n          b.getNumBytes() % cacheManager.getOsPageSize() == 0 &&\n          reserveLockedMemory(b.getNumBytes())) {\n        try {\n          // First try to place the block on a transient volume.\n          ref = volumes.getNextTransientVolume(b.getNumBytes());\n          datanode.getMetrics().incrRamDiskBlocksWrite();","sourceCodeStart":1615,"sourceCodeEnd":1651,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L1615-L1651","documentation":"When a client asks a DataNode to create a brand-new RBW replica, an existing volumeMap entry for the same blockPoolId+blockId is only acceptable if the request is a retry carrying a newer generation stamp (newGS != 0, which triggers cleanupReplica of the old copy). With newGS == 0 - a first-time create - any pre-existing replica is a contradiction, and ReplicaAlreadyExistsException is thrown.","triggerScenarios":"BlockReceiver.java:221 createRbw(storageType, storageId, b, allowLazyPersist, newGS) for a fresh block whose id already has a replica in this DN's volumeMap: an earlier abandoned create left a replica behind, and the retry arrived without a bumped GS.","commonSituations":"Pipeline setup failed on another DN and the retry re-lands on this DN before the client bumps the GS; stale volumeMap entries after DN restart with leftover rbw files; very rarely, duplicate block allocation by the NN.","solutions":["Retry the create with a bumped GS - with newGS != 0 the DataNode cleans up the old replica itself (this is the designed path; DFSOutputStream requests a new GS when it catches this exception)","If retries loop, inspect the DN's rbw/finalized directories for the blockId and remove the stray replica after fsck confirms no valid copy needs it","Run 'hdfs fsck -files -blocks -locations' to make sure the blockId is not double-allocated","Check DN logs for the earlier create attempt that left the replica behind"],"exampleFix":"// before: retrying createRbw with newGS=0 forever\nfsDataset.createRbw(storageType, storageId, b, allowLazyPersist, 0L);\n// -> ReplicaAlreadyExistsException: Block ... already exists ...\n\n// after: on 'already exists', bump the GS so the DN cleans the old replica\ntry {\n  fsDataset.createRbw(storageType, storageId, b, allowLazyPersist, 0L);\n} catch (ReplicaAlreadyExistsException e) {\n  long bumpedGs = b.getGenerationStamp() + 1;\n  ExtendedBlock nb = new ExtendedBlock(\n      b.getBlockPoolId(), b.getBlockId(), b.getNumBytes(), bumpedGs);\n  fsDataset.createRbw(storageType, storageId, nb, allowLazyPersist, bumpedGs);\n}","handlingStrategy":"retry","validationCode":"Replica existing = fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());\nif (existing != null && newGS == 0L) {\n  // a first-time create cannot replace an existing replica: caller must bump the GS\n  throw new IllegalStateException(\n      \"replica already exists for \" + b + \"; retry createRbw with a non-zero newGS\");\n}\nfsDataset.createRbw(storageType, storageId, b, allowLazyPersist, newGS);","typeGuard":null,"tryCatchPattern":"catch (ReplicaAlreadyExistsException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"already exists\")) {\n    long bumpedGs = b.getGenerationStamp() + 1;\n    retryCreateRbwWithNewGS(bumpedGs); // newGS != 0 makes the DN clean the old replica\n  } else { throw e; }\n}","preventionTips":["Make block creates idempotent: carry a fresh GS on every retry so stale replicas are cleaned, not collided with","Do not cache and replay OP_WRITE_BLOCK pipeline setups verbatim after failure","Clean stray rbw files in DN storage directories when rebuilding or reusing a DataNode's data"],"tags":["hdfs","datanode","block-write","create-rbw","idempotency"],"backgroundTag":"hdfs-block-already-exists","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}