apache/hadoop · error · IOException

Receive reply from slowNode {} for continuous {} times, trea

Error message

Receive reply from slowNode {} for continuous {} times, treating it as badNode

What it means

Error "Receive reply from slowNode {} for continuous {} times, treating it as badNode" thrown in apache/hadoop.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java:1338

          slowNodeMap.put(slowNode, 1);
        } else {
          int oldCount = slowNodeMap.get(slowNode);
          slowNodeMap.put(slowNode, ++oldCount);
        }
        discontinuousNodes.remove(slowNode);
      }
      for (DatanodeInfo discontinuousNode : discontinuousNodes) {
        slowNodeMap.remove(discontinuousNode);
      }

      if (!slowNodeMap.isEmpty()) {
        for (Map.Entry<DatanodeInfo, Integer> entry : slowNodeMap.entrySet()) {
          if (entry.getValue() >= markSlowNodeAsBadNodeThreshold) {
            DatanodeInfo slowNode = entry.getKey();
            int index = getDatanodeIndex(slowNode);
            if (index >= 0) {
              errorState.setBadNodeIndex(index);
              throw new IOException("Receive reply from slowNode " + slowNode +
                  " for continuous " + markSlowNodeAsBadNodeThreshold +
                  " times, treating it as badNode");
            }
            slowNodeMap.remove(entry.getKey());
          }
        }
      }
    }

    void close() {
      responderClosed = true;
      this.interrupt();
    }

    int getDatanodeIndex(DatanodeInfo datanodeInfo) {
      for (int i = 0; i < targets.length; i++) {
        if (targets[i].equals(datanodeInfo)) {
          return i;

View on GitHub (pinned to 2add963021)

Solutions

1. Check network and DataNode load; the slow DataNode was marked bad after repeated slow responses, so the write pipeline will be re-established without it.
2. Increase dfs.client.slow.io.threshold or investigate DataNode disk/GC pauses if slow nodes are reported frequently.
3. Verify cluster health with hdfs dfsadmin -report and decommission or fix consistently slow DataNodes.

When it happens

Trigger: A DataNode in the write pipeline responds slowly on consecutive acknowledgments. After the configured slow-node threshold is reached, the client marks the node as bad and removes it from the pipeline.

Common situations: See trigger scenarios.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/be9fc254a052411c. Report an issue: GitHub.