apache/hadoop · error · IOException

Datanode {} is restarting: {}

Error message

Datanode {} is restarting: {}

What it means

While processing pipeline acks, DataStreamer's ResponseProcessor recognizes the out-of-band RESTART status: a datanode is shutting down for a restart (often a rolling upgrade) and tells writers so they can plan. The client records the restarting node in errorState (initRestartingNode, with waiting when it is the local node or the only pipeline member) and throws IOException('Datanode i is restarting: <node>') to drive pipeline handling.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java:1218

          for (int i = ack.getNumOfReplies()-1; i >=0  && dfsClient.clientRunning; i--) {
            final Status reply = PipelineAck.getStatusFromHeader(ack
                .getHeaderFlag(i));
            if (PipelineAck.getECNFromHeader(ack.getHeaderFlag(i)) ==
                PipelineAck.ECN.CONGESTED) {
              congestedNodesFromAck.add(targets[i]);
            }
            if (PipelineAck.getSLOWFromHeader(ack.getHeaderFlag(i)) ==
                PipelineAck.SLOW.SLOW) {
              slownodesFromAck.add(targets[i]);
            }
            // Restart will not be treated differently unless it is
            // the local node or the only one in the pipeline.
            if (PipelineAck.isRestartOOBStatus(reply)) {
              final String message = "Datanode " + i + " is restarting: "
                  + targets[i];
              errorState.initRestartingNode(i, message,
                  shouldWaitForRestart(i));
              throw new IOException(message);
            }
            // node error
            if (reply != SUCCESS) {
              errorState.setBadNodeIndex(i); // mark bad datanode
              throw new IOException("Bad response " + reply +
                  " for " + block + " from datanode " + targets[i]);
            }
          }

          if (!congestedNodesFromAck.isEmpty()) {
            synchronized (congestedNodes) {
              congestedNodes.clear();
              congestedNodes.addAll(congestedNodesFromAck);
            }
          } else {
            synchronized (congestedNodes) {
              congestedNodes.clear();
              lastCongestionBackoffTime = 0;

View on GitHub (pinned to 2add963021)

Solutions

1. Treat as transient: DataStreamer rebuilds the pipeline; retry the operation if it still surfaces to the application.
2. Schedule datanode restarts outside write-heavy windows or drain writers first.
3. If writes stall, confirm the restarting DN came back and registered (hdfs dfsadmin -report) — the client waits only when that node is local or the only one in the pipeline.
4. Avoid lowering datanodeRestartTimeout-related settings on clusters with frequent planned restarts.

try {
  out.write(buf, 0, len);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("is restarting")) {
    // transient during rolling upgrade/restart: back off and retry;
    // DataStreamer already waits for local/sole-node restarts up to its timeout
    sleepBackoff();
    out.write(buf, 0, len);
  } else {
    throw e;
  }
}

When it happens

Trigger: Writing while a pipeline datanode restarts: the DN sends the OOB restart ack instead of SUCCESS. The client may wait for the restart (shouldWaitForRestart: local node or single-node pipeline) before rebuilding the pipeline; if waiting exceeds datanodeRestartTimeout, recovery proceeds without the node.

Common situations: Rolling upgrades or planned maintenance during active writes; automatic DN restart on upgrade; co-located clients (local DN restart forces the wait path); CI clusters restarted on a schedule under write load.

Related errors

• Unable to create new block.{}
• File %s could only be written to %d of the %d %s. There are
• Missing storageIDs: It is likely that the HDFS client, who m
• The \"downgrade\" option is no longer supported since it may
• Failed to convert \"{s}\" to RollingUpgradeStartupOption

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/165ccd29f623be29. Report an issue: GitHub.