{"record":{"id":"3742d5463692458c","repo":"apache/hadoop","slug":"interrupted-while-choosing-datanode-for-read","errorCode":null,"errorMessage":"Interrupted while choosing DataNode for read.","messagePattern":"Interrupted while choosing DataNode for read\\.","errorType":"exception","errorClass":"InterruptedIOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java","lineNumber":1050,"sourceCode":"      // At the first time of getting a BlockMissingException, the wait time\n      // is a random number between 0..3000 ms. If the first retry\n      // still fails, we will wait 3000 ms grace period before the 2nd retry.\n      // Also at the second retry, the waiting window is expanded to 6000 ms\n      // alleviating the request rate from the server. Similarly the 3rd retry\n      // will wait 6000ms grace period before retry and the waiting window is\n      // expanded to 9000ms.\n      final int timeWindow = dfsClient.getConf().getTimeWindow();\n      // grace period for the last round of attempt\n      double waitTime = timeWindow * failures +\n          // expanding time window for each failure\n          timeWindow * (failures + 1) *\n          ThreadLocalRandom.current().nextDouble();\n      DFSClient.LOG.warn(\"DFS chooseDataNode: got # \" + (failures + 1) +\n          \" IOException, will wait for \" + waitTime + \" msec.\");\n      Thread.sleep((long)waitTime);\n    } catch (InterruptedException e) {\n      Thread.currentThread().interrupt();\n      throw new InterruptedIOException(\n          \"Interrupted while choosing DataNode for read.\");\n    }\n    clearCachedNodeState(ignoredNodes);\n    openInfo(true);\n    block = refreshLocatedBlock(block);\n    failures++;\n    return block;\n  }\n\n  /**\n   * Clear both the dead nodes and the ignored nodes\n   * @param ignoredNodes is cleared\n   */\n  private void clearCachedNodeState(Collection<DatanodeInfo> ignoredNodes) {\n    clearLocalDeadNodes(); //2nd option is to remove only nodes[blockId]\n    if (ignoredNodes != null) {\n      ignoredNodes.clear();\n    }","sourceCodeStart":1032,"sourceCodeEnd":1068,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java#L1032-L1068","documentation":"While choosing a DataNode after failures, the client sleeps an exponentially growing, randomized backoff (timeWindow * failures plus a random factor up to timeWindow*(failures+1)). Interrupting the thread during that sleep converts to InterruptedIOException with the interrupt flag restored, aborting the node selection and the read.","triggerScenarios":"Thread interrupt during the retry backoff between DN attempts - i.e. the read already saw DN failures, then the thread was cancelled mid-wait.","commonSituations":"Task cancellation exactly when a struggling read (dead DNs) is backing off; aggressive executors calling shutdownNow() on reader pools.","solutions":["Honor the cancellation: unwind the reader instead of retrying","Cancel HDFS reads via stream close, not thread interrupt","Keep DN failure rates low (fix the underlying node/network issues) so this backoff path is rarely active when interrupts land"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":"static boolean isCancelled(IOException e) {\n  return e instanceof InterruptedIOException;\n}","tryCatchPattern":"try {\n  n = in.read(buf);\n} catch (InterruptedIOException e) {\n  // cancelled during DN-failure backoff: unwind, do not retry\n  throw new java.util.concurrent.CancellationException(\"read interrupted\", e);\n}","preventionTips":["Cancel HDFS reads via close(), never via Thread.interrupt()","Keep DN/network healthy so the backoff path is rarely active when interrupts occur"],"tags":["hdfs","interrupt","datanode","backoff"],"backgroundTag":"thread-interrupted","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}