{"record":{"id":"473df0cb1b8449cf","repo":"apache/hadoop","slug":"interrupted-while-running-disk-check","errorCode":null,"errorMessage":"Interrupted while running disk check","messagePattern":"Interrupted while running disk check","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java","lineNumber":4080,"sourceCode":"    return ecReconstuctReadThrottler;\n  }\n\n  public DataTransferThrottler getEcReconstuctWriteThrottler() {\n    return ecReconstuctWriteThrottler;\n  }\n\n  /**\n   * Check the disk error synchronously.\n   */\n  @VisibleForTesting\n  public void checkDiskError() throws IOException {\n    Set<FsVolumeSpi> unhealthyVolumes;\n    try {\n      unhealthyVolumes = volumeChecker.checkAllVolumes(data);\n      lastDiskErrorCheck = Time.monotonicNow();\n    } catch (InterruptedException e) {\n      LOG.error(\"Interrupted while running disk check\", e);\n      throw new IOException(\"Interrupted while running disk check\", e);\n    }\n\n    if (unhealthyVolumes.size() > 0) {\n      LOG.warn(\"checkDiskError got {} failed volumes - {}\",\n          unhealthyVolumes.size(), unhealthyVolumes);\n      handleVolumeFailures(unhealthyVolumes);\n    } else {\n      LOG.debug(\"checkDiskError encountered no failures\");\n    }\n  }\n\n  @VisibleForTesting\n  public void handleVolumeFailures(Set<FsVolumeSpi> unhealthyVolumes) {\n    if (unhealthyVolumes.isEmpty()) {\n      LOG.debug(\"handleVolumeFailures done with empty \" +\n          \"unhealthyVolumes\");\n      return;\n    }","sourceCodeStart":4062,"sourceCodeEnd":4098,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L4062-L4098","documentation":"checkDiskError runs the volume checker synchronously over all FsVolumes. If the calling thread is interrupted while volumeChecker.checkAllVolumes(data) blocks, the InterruptedException is logged and rethrown wrapped in IOException so callers see one checked type. This signals thread interruption (usually DN shutdown), not a disk fault.","triggerScenarios":"The thread executing DataNode.checkDiskError() is interrupted - datanode shutdown while a synchronous disk/volume check is in flight, or external tooling cancelling DN worker threads during a long-running checkAllVolumes call.","commonSituations":"DN shutdown overlapping slow disk checks (many volumes or slow disks); JVM shutdown hooks interrupting executor threads; tests or agents that interrupt DataNode threads mid-check.","solutions":["Restore the interrupt status (Thread.currentThread().interrupt()) and let the datanode shut down cleanly - do not retry the check on an interrupted thread","If this fires outside shutdown, find what interrupts DN threads (management agents, container runtime signals)","Tune disk-check timeouts (dfs.datanode.disk.check.timeout and similar) if checks regularly overlap shutdown","Verify disk health separately (smartctl/iostat) since the check was aborted, not completed"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  datanode.checkDiskError();\n} catch (IOException e) {\n  if (e.getCause() instanceof InterruptedException) {\n    Thread.currentThread().interrupt(); // restore flag, abort gracefully\n    return;\n  }\n  throw e;\n}","preventionTips":["Never interrupt DN threads that may be inside volume checks; use the DN's own shutdown path","Keep disk checks short enough (timeouts, number of volumes) that they do not overlap shutdown","When this fires during shutdown, treat it as expected noise - do not retry the disk check"],"tags":["hadoop","hdfs","datanode","interrupted","disk-check","concurrency"],"backgroundTag":"thread-interrupted","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}