apache/hadoop · error · IOException

Retry " + retry + " times to read still exception: " + error

Error message

Retry " + retry + " times to read still exception: " + errorMsg

What it means

BosInputStream.read retries a failed read with exponential backoff (intervalSeconds doubles, retry decrements). When the counter hits 0 and the read still throws, the last exception is wrapped in IOException 'Retry N times to read still exception: <cause>'. Note the message prints the exhausted counter (0), not how many attempts were made, and the real failure is the wrapped cause 'ex', not this wrapper.

Source

Thrown at hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BosInputStream.java:210

          break;
        } catch (EOFException eof) {
          return -1;
        } catch (IOException ioe) {
          // For IOException, try to reopen once more
          // and retry
          LOG.info(
              "IOException during retry,"
                  + " attempting to reopen stream");
          onReadFailure(ioe, len);
          bytesRead = in.read(buf, off, len);
          break;
        } catch (Exception ex) {
          if (retry <= 0) {
            String errorMsg =
                ex.getMessage() != null
                    ? ex.getMessage()
                    : ex.getClass().getSimpleName();
            throw new IOException(
                "Retry " + retry
                    + " times to read still"
                    + " exception: " + errorMsg,
                ex);
          }

          try {
            TimeUnit.SECONDS.sleep(
                intervalSeconds);
          } catch (InterruptedException ite) {
            Thread.currentThread().interrupt();
            throw new IOException(
                "Thread interrupted during retry", ite);
          }
          intervalSeconds *= 2;
          retry--;
          // Update e to ex for the next iteration
          e = ex;

View on GitHub (pinned to 2add963021)

Solutions

Read getCause() of the thrown IOException first — it holds the actual last failure and dictates the fix
If outages are short, raise the input-stream retry count and initial interval so backoff covers the blip
For long tasks, refresh STS/session tokens before expiry or use longer-lived credentials
At task level, catch this and reopen the stream from the last checkpointed position instead of failing the job
Verify network reachability of the BOS endpoint from the failing node

Example fix

// before: defaults
fs.bos.input.stream.retry=3
fs.bos.input.stream.retry.interval.seconds=1

// after: more attempts with exponential backoff
fs.bos.input.stream.retry=6
fs.bos.input.stream.retry.interval.seconds=2

Defensive patterns

Strategy: retry

Type guard

static boolean isRetryExhaustedRead(IOException e) {
  return e.getMessage() != null && e.getMessage().startsWith("Retry ")
      && e.getMessage().contains("still exception");
}

Try / catch

catch (IOException e) {
  if (isRetryExhaustedRead(e) && e.getCause() != null) {
    LOG.error("underlying failure", e.getCause()); // diagnose the cause, not the wrapper
  }
  in = fs.open(path); // full reopen from scratch
  in.seek(lastGoodPos); // resume from checkpoint
}

Prevention

Checkpoint read positions so a failed stream can be reopened and resumed
Size the retry budget to your network's worst blip
Use credentials whose TTL exceeds the longest task

When it happens

Trigger: Persistent read failure that survives every internal retry: network outage to the BOS endpoint, expired STS/session token, object deleted or truncated mid-read, or sustained 429/5xx on every reopen attempt.

Common situations: Long-running MR/Spark tasks crossing short network blips with too small a retry budget; session tokens expiring on very long tasks; reading an object that a concurrent writer overwrites or deletes; misconfigured endpoint/DNS.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/64a257c1969e5f97. Report an issue: GitHub.