What causes "Thread interrupted during retry" in apache/hadoop?

Treat it as cancellation, not corruption: let it propagate and make sure the task exits promptly; do not 'fix' by clearing the interrupt

apache/hadoop · warning · IOException

Thread interrupted during retry

Error message

Thread interrupted during retry

What it means

While sleeping between read retries, BosInputStream is interrupted; it re-asserts the interrupt flag (Thread.currentThread().interrupt()) and throws IOException('Thread interrupted during retry') wrapping the InterruptedException. This converts task cancellation into an I/O failure so upper layers unwind cleanly.

Source

Thrown at hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BosInputStream.java:222

        } catch (Exception ex) {
          if (retry <= 0) {
            String errorMsg =
                ex.getMessage() != null
                    ? ex.getMessage()
                    : ex.getClass().getSimpleName();
            throw new IOException(
                "Retry " + retry
                    + " times to read still"
                    + " exception: " + errorMsg,
                ex);
          }

          try {
            TimeUnit.SECONDS.sleep(
                intervalSeconds);
          } catch (InterruptedException ite) {
            Thread.currentThread().interrupt();
            throw new IOException(
                "Thread interrupted during retry", ite);
          }
          intervalSeconds *= 2;
          retry--;
          // Update e to ex for the next iteration
          e = ex;
        }
      }
    }

    if (bytesRead > 0) {
      pos += bytesRead;
      nextReadPos += bytesRead;
    }

    if (statistics != null && bytesRead >= 0) {
      statistics.incrementBytesRead(bytesRead);
    }

View on GitHub (pinned to 2add963021)

Solutions

Treat it as cancellation, not corruption: let it propagate and make sure the task exits promptly; do not 'fix' by clearing the interrupt
If spurious speculative kills are common, disable mapreduce.map.speculative for these jobs or fix the slow-task cause
Shorten backoff windows so tasks respond to cancellation faster

Defensive patterns

Strategy: try-catch

Type guard

static boolean isCancellation(IOException e) {
  return "Thread interrupted during retry".equals(e.getMessage())
      && e.getCause() instanceof InterruptedException;
}

Try / catch

catch (IOException e) {
  if (isCancellation(e)) {
    // cooperative shutdown: stop the reader promptly, keep the interrupt status
    throw new RuntimeException("cancelled", e);
  }
  throw e;
}

Prevention

Never swallow the interrupt — honor cancellation semantics
Keep backoff windows bounded so tasks die quickly when killed
Disable speculative execution if duplicate tasks churn on slow inputs

When it happens

Trigger: The reading thread is interrupted during the backoff window: Hadoop kills a speculative duplicate task, the job is cancelled, a Spark task is retracted, or an executor/thread-pool shuts down mid-retry.

Common situations: Speculative execution killing slow-but-healthy mappers that happen to be backing off; user job cancellation during a BOS outage; application shutdown racing an in-flight retry loop.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/be67f088969efe90. Report an issue: GitHub.