apache/hadoop · error · InterruptedIOException

Interrupted while choosing DataNode for read.

Error message

Interrupted while choosing DataNode for read.

What it means

While choosing a DataNode after failures, the client sleeps an exponentially growing, randomized backoff (timeWindow * failures plus a random factor up to timeWindow*(failures+1)). Interrupting the thread during that sleep converts to InterruptedIOException with the interrupt flag restored, aborting the node selection and the read.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java:1050

      // At the first time of getting a BlockMissingException, the wait time
      // is a random number between 0..3000 ms. If the first retry
      // still fails, we will wait 3000 ms grace period before the 2nd retry.
      // Also at the second retry, the waiting window is expanded to 6000 ms
      // alleviating the request rate from the server. Similarly the 3rd retry
      // will wait 6000ms grace period before retry and the waiting window is
      // expanded to 9000ms.
      final int timeWindow = dfsClient.getConf().getTimeWindow();
      // grace period for the last round of attempt
      double waitTime = timeWindow * failures +
          // expanding time window for each failure
          timeWindow * (failures + 1) *
          ThreadLocalRandom.current().nextDouble();
      DFSClient.LOG.warn("DFS chooseDataNode: got # " + (failures + 1) +
          " IOException, will wait for " + waitTime + " msec.");
      Thread.sleep((long)waitTime);
    } catch (InterruptedException e) {
      Thread.currentThread().interrupt();
      throw new InterruptedIOException(
          "Interrupted while choosing DataNode for read.");
    }
    clearCachedNodeState(ignoredNodes);
    openInfo(true);
    block = refreshLocatedBlock(block);
    failures++;
    return block;
  }

  /**
   * Clear both the dead nodes and the ignored nodes
   * @param ignoredNodes is cleared
   */
  private void clearCachedNodeState(Collection<DatanodeInfo> ignoredNodes) {
    clearLocalDeadNodes(); //2nd option is to remove only nodes[blockId]
    if (ignoredNodes != null) {
      ignoredNodes.clear();
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Honor the cancellation: unwind the reader instead of retrying
2. Cancel HDFS reads via stream close, not thread interrupt
3. Keep DN failure rates low (fix the underlying node/network issues) so this backoff path is rarely active when interrupts land

static boolean isCancelled(IOException e) {
  return e instanceof InterruptedIOException;
}

try {
  n = in.read(buf);
} catch (InterruptedIOException e) {
  // cancelled during DN-failure backoff: unwind, do not retry
  throw new java.util.concurrent.CancellationException("read interrupted", e);
}

When it happens

Trigger: Thread interrupt during the retry backoff between DN attempts - i.e. the read already saw DN failures, then the thread was cancelled mid-wait.

Common situations: Task cancellation exactly when a struggling read (dead DNs) is backing off; aggressive executors calling shutdownNow() on reader pools.

Related errors

• Version Mismatch (Expected: {}, Received: {} )
• Unknown op {} in data stream
• Cannot create a secured connection if DataNode listens on un
• Received %x instead of %x from client.
• Replica was found but missing fields.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3742d5463692458c. Report an issue: GitHub.