apache/hadoop · error · IOException

Unable to stop existing writer for block {b} after {writerSt

Error message

Unable to stop existing writer for block {b} after {writerStopMs} miniseconds.

What it means

IOException (preceded by a LOG.warn with the same text) thrown by FsDatasetImpl.createTemporary when an existing TEMPORARY/RBW replica's writer thread cannot be stopped within dfs.datanode.xceiver.stop.timeout.millis (default 60000). The loop stops the previous writer via stopWriter() to take over the block, and bails out when the writer hangs. The comment in code says this is not supposed to happen.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:1919

          if (((currentReplicaInfo.getGenerationStamp() >= b
              .getGenerationStamp()) || (!isTransfer && !isInPipeline))
              && !isReplicaProvided(currentReplicaInfo)) {
            throw new ReplicaAlreadyExistsException("Block " + b
                + " already exists in state " + currentReplicaInfo.getState()
                + " and thus cannot be created.");
          }
          lastFoundReplicaInfo = currentReplicaInfo;
        }
      }
      if (!isInPipeline) {
        continue;
      }
      // Hang too long, just bail out. This is not supposed to happen.
      long writerStopMs = Time.monotonicNow() - startTimeMs;
      if (writerStopMs > writerStopTimeoutMs) {
        LOG.warn("Unable to stop existing writer for block " + b + " after " 
            + writerStopMs + " miniseconds.");
        throw new IOException("Unable to stop existing writer for block " + b
            + " after " + writerStopMs + " miniseconds.");
      }

      // if lastFoundReplicaInfo is PROVIDED and FINALIZED,
      // stopWriter isn't required.
      if (isReplicaProvided(lastFoundReplicaInfo) &&
          lastFoundReplicaInfo.getState() == ReplicaState.FINALIZED) {
        continue;
      }
      // Stop the previous writer
      ((ReplicaInPipeline)lastFoundReplicaInfo).stopWriter(writerStopTimeoutMs);
    } while (true);
    long holdLockTimeMs = Time.monotonicNow() - startTimeMs;
    if (lastFoundReplicaInfo != null
        && !isReplicaProvided(lastFoundReplicaInfo)) {
      // Old blockfile should be deleted synchronously as it might collide
      // with the new block if allocated in same volume.
      // Do the deletion outside of lock as its DISK IO.

View on GitHub (pinned to 2add963021)

Solutions

Check DataNode disk health (smartctl, dmesg for I/O errors) - a hung writer is most often stuck on disk I/O.
Increase dfs.datanode.xceiver.stop.timeout.millis if writers legitimately need longer (e.g., slow but healthy disks under heavy load).
Restart the DataNode to clear the stuck writer thread; the block will be recovered through normal lease recovery.
Review GC logs for multi-second pauses and tune heap if pauses are the cause.

Defensive patterns

Strategy: retry

Try / catch

try {
  dataset.createTemporary(storageType, storageId, b, isTransfer);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Unable to stop existing writer")) {
    // writer hang: raise a disk-health alert; retry after the stuck writer is cleared (often needs DN restart)
    diskHealthAlert(e);
  }
  throw e;
}

Prevention

Set dfs.datanode.xceiver.stop.timeout.millis above your worst-case legitimate flush time (default 60000 ms).
Monitor DataNode disk latency and xceiver thread counts so hung writers surface early.
Keep dfs.data.dir off network filesystems; local disks only.
Watch GC pause times on DataNodes; long pauses make interruptible writers miss deadlines.

When it happens

Trigger: createTemporary for a block whose replica is currently being written by another thread; that writer thread ignores the interrupt and the timeout set via ReplicaInPipeline.stopWriter(writerStopTimeoutMs) expires, so Time.monotonicNow() - startTimeMs > writerStopTimeoutMs.

Common situations: A DataXceiver thread stuck in a slow/hung disk write (failing disk, NFS-mounted dfs.data.dir, overloaded storage); long GC pauses on the DataNode; extremely slow finalize/fsync during concurrent lease recovery; timeout configured too low for the workload.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8752be395bc6c769. Report an issue: GitHub.