{"record":{"id":"8752be395bc6c769","repo":"apache/hadoop","slug":"unable-to-stop-existing-writer-for-block-b-after","errorCode":null,"errorMessage":"Unable to stop existing writer for block {b} after {writerStopMs} miniseconds.","messagePattern":"Unable to stop existing writer for block (.+?) after (.+?) miniseconds\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":1919,"sourceCode":"          if (((currentReplicaInfo.getGenerationStamp() >= b\n              .getGenerationStamp()) || (!isTransfer && !isInPipeline))\n              && !isReplicaProvided(currentReplicaInfo)) {\n            throw new ReplicaAlreadyExistsException(\"Block \" + b\n                + \" already exists in state \" + currentReplicaInfo.getState()\n                + \" and thus cannot be created.\");\n          }\n          lastFoundReplicaInfo = currentReplicaInfo;\n        }\n      }\n      if (!isInPipeline) {\n        continue;\n      }\n      // Hang too long, just bail out. This is not supposed to happen.\n      long writerStopMs = Time.monotonicNow() - startTimeMs;\n      if (writerStopMs > writerStopTimeoutMs) {\n        LOG.warn(\"Unable to stop existing writer for block \" + b + \" after \" \n            + writerStopMs + \" miniseconds.\");\n        throw new IOException(\"Unable to stop existing writer for block \" + b\n            + \" after \" + writerStopMs + \" miniseconds.\");\n      }\n\n      // if lastFoundReplicaInfo is PROVIDED and FINALIZED,\n      // stopWriter isn't required.\n      if (isReplicaProvided(lastFoundReplicaInfo) &&\n          lastFoundReplicaInfo.getState() == ReplicaState.FINALIZED) {\n        continue;\n      }\n      // Stop the previous writer\n      ((ReplicaInPipeline)lastFoundReplicaInfo).stopWriter(writerStopTimeoutMs);\n    } while (true);\n    long holdLockTimeMs = Time.monotonicNow() - startTimeMs;\n    if (lastFoundReplicaInfo != null\n        && !isReplicaProvided(lastFoundReplicaInfo)) {\n      // Old blockfile should be deleted synchronously as it might collide\n      // with the new block if allocated in same volume.\n      // Do the deletion outside of lock as its DISK IO.","sourceCodeStart":1901,"sourceCodeEnd":1937,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L1901-L1937","documentation":"IOException (preceded by a LOG.warn with the same text) thrown by FsDatasetImpl.createTemporary when an existing TEMPORARY/RBW replica's writer thread cannot be stopped within dfs.datanode.xceiver.stop.timeout.millis (default 60000). The loop stops the previous writer via stopWriter() to take over the block, and bails out when the writer hangs. The comment in code says this is not supposed to happen.","triggerScenarios":"createTemporary for a block whose replica is currently being written by another thread; that writer thread ignores the interrupt and the timeout set via ReplicaInPipeline.stopWriter(writerStopTimeoutMs) expires, so Time.monotonicNow() - startTimeMs > writerStopTimeoutMs.","commonSituations":"A DataXceiver thread stuck in a slow/hung disk write (failing disk, NFS-mounted dfs.data.dir, overloaded storage); long GC pauses on the DataNode; extremely slow finalize/fsync during concurrent lease recovery; timeout configured too low for the workload.","solutions":["Check DataNode disk health (smartctl, dmesg for I/O errors) - a hung writer is most often stuck on disk I/O.","Increase dfs.datanode.xceiver.stop.timeout.millis if writers legitimately need longer (e.g., slow but healthy disks under heavy load).","Restart the DataNode to clear the stuck writer thread; the block will be recovered through normal lease recovery.","Review GC logs for multi-second pauses and tune heap if pauses are the cause."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  dataset.createTemporary(storageType, storageId, b, isTransfer);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Unable to stop existing writer\")) {\n    // writer hang: raise a disk-health alert; retry after the stuck writer is cleared (often needs DN restart)\n    diskHealthAlert(e);\n  }\n  throw e;\n}","preventionTips":["Set dfs.datanode.xceiver.stop.timeout.millis above your worst-case legitimate flush time (default 60000 ms).","Monitor DataNode disk latency and xceiver thread counts so hung writers surface early.","Keep dfs.data.dir off network filesystems; local disks only.","Watch GC pause times on DataNodes; long pauses make interruptible writers miss deadlines."],"tags":["hdfs","datanode","writer-stop","timeout","disk-io"],"backgroundTag":"writer-stop-timeout","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}