What causes "Interrupted while running disk check" in apache/hadoop?

Restore the interrupt status (Thread.currentThread().interrupt()) and let the datanode shut down cleanly - do not retry the check on an interrupted thread

apache/hadoop · error · IOException

Interrupted while running disk check

Error message

Interrupted while running disk check

What it means

checkDiskError runs the volume checker synchronously over all FsVolumes. If the calling thread is interrupted while volumeChecker.checkAllVolumes(data) blocks, the InterruptedException is logged and rethrown wrapped in IOException so callers see one checked type. This signals thread interruption (usually DN shutdown), not a disk fault.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:4080

    return ecReconstuctReadThrottler;
  }

  public DataTransferThrottler getEcReconstuctWriteThrottler() {
    return ecReconstuctWriteThrottler;
  }

  /**
   * Check the disk error synchronously.
   */
  @VisibleForTesting
  public void checkDiskError() throws IOException {
    Set<FsVolumeSpi> unhealthyVolumes;
    try {
      unhealthyVolumes = volumeChecker.checkAllVolumes(data);
      lastDiskErrorCheck = Time.monotonicNow();
    } catch (InterruptedException e) {
      LOG.error("Interrupted while running disk check", e);
      throw new IOException("Interrupted while running disk check", e);
    }

    if (unhealthyVolumes.size() > 0) {
      LOG.warn("checkDiskError got {} failed volumes - {}",
          unhealthyVolumes.size(), unhealthyVolumes);
      handleVolumeFailures(unhealthyVolumes);
    } else {
      LOG.debug("checkDiskError encountered no failures");
    }
  }

  @VisibleForTesting
  public void handleVolumeFailures(Set<FsVolumeSpi> unhealthyVolumes) {
    if (unhealthyVolumes.isEmpty()) {
      LOG.debug("handleVolumeFailures done with empty " +
          "unhealthyVolumes");
      return;
    }

View on GitHub (pinned to 2add963021)

Solutions

Restore the interrupt status (Thread.currentThread().interrupt()) and let the datanode shut down cleanly - do not retry the check on an interrupted thread
If this fires outside shutdown, find what interrupts DN threads (management agents, container runtime signals)
Tune disk-check timeouts (dfs.datanode.disk.check.timeout and similar) if checks regularly overlap shutdown
Verify disk health separately (smartctl/iostat) since the check was aborted, not completed

Defensive patterns

Strategy: try-catch

Try / catch

try {
  datanode.checkDiskError();
} catch (IOException e) {
  if (e.getCause() instanceof InterruptedException) {
    Thread.currentThread().interrupt(); // restore flag, abort gracefully
    return;
  }
  throw e;
}

Prevention

Never interrupt DN threads that may be inside volume checks; use the DN's own shutdown path
Keep disk checks short enough (timeouts, number of volumes) that they do not overlap shutdown
When this fires during shutdown, treat it as expected noise - do not retry the disk check

When it happens

Trigger: The thread executing DataNode.checkDiskError() is interrupted - datanode shutdown while a synchronous disk/volume check is in flight, or external tooling cancelling DN worker threads during a long-running checkAllVolumes call.

Common situations: DN shutdown overlapping slow disk checks (many volumes or slow disks); JVM shutdown hooks interrupting executor threads; tests or agents that interrupt DataNode threads mid-check.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/473df0cb1b8449cf. Report an issue: GitHub.