{"record":{"id":"64a257c1969e5f97","repo":"apache/hadoop","slug":"retry-retry-times-to-read-still-exception","errorCode":null,"errorMessage":"Retry \" + retry + \" times to read still exception: \" + errorMsg","messagePattern":"Retry \" \\+ retry \\+ \" times to read still exception: \" \\+ errorMsg","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BosInputStream.java","lineNumber":210,"sourceCode":"          break;\n        } catch (EOFException eof) {\n          return -1;\n        } catch (IOException ioe) {\n          // For IOException, try to reopen once more\n          // and retry\n          LOG.info(\n              \"IOException during retry,\"\n                  + \" attempting to reopen stream\");\n          onReadFailure(ioe, len);\n          bytesRead = in.read(buf, off, len);\n          break;\n        } catch (Exception ex) {\n          if (retry <= 0) {\n            String errorMsg =\n                ex.getMessage() != null\n                    ? ex.getMessage()\n                    : ex.getClass().getSimpleName();\n            throw new IOException(\n                \"Retry \" + retry\n                    + \" times to read still\"\n                    + \" exception: \" + errorMsg,\n                ex);\n          }\n\n          try {\n            TimeUnit.SECONDS.sleep(\n                intervalSeconds);\n          } catch (InterruptedException ite) {\n            Thread.currentThread().interrupt();\n            throw new IOException(\n                \"Thread interrupted during retry\", ite);\n          }\n          intervalSeconds *= 2;\n          retry--;\n          // Update e to ex for the next iteration\n          e = ex;","sourceCodeStart":192,"sourceCodeEnd":228,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BosInputStream.java#L192-L228","documentation":"BosInputStream.read retries a failed read with exponential backoff (intervalSeconds doubles, retry decrements). When the counter hits 0 and the read still throws, the last exception is wrapped in IOException 'Retry N times to read still exception: <cause>'. Note the message prints the exhausted counter (0), not how many attempts were made, and the real failure is the wrapped cause 'ex', not this wrapper.","triggerScenarios":"Persistent read failure that survives every internal retry: network outage to the BOS endpoint, expired STS/session token, object deleted or truncated mid-read, or sustained 429/5xx on every reopen attempt.","commonSituations":"Long-running MR/Spark tasks crossing short network blips with too small a retry budget; session tokens expiring on very long tasks; reading an object that a concurrent writer overwrites or deletes; misconfigured endpoint/DNS.","solutions":["Read getCause() of the thrown IOException first — it holds the actual last failure and dictates the fix","If outages are short, raise the input-stream retry count and initial interval so backoff covers the blip","For long tasks, refresh STS/session tokens before expiry or use longer-lived credentials","At task level, catch this and reopen the stream from the last checkpointed position instead of failing the job","Verify network reachability of the BOS endpoint from the failing node"],"exampleFix":"// before: defaults\nfs.bos.input.stream.retry=3\nfs.bos.input.stream.retry.interval.seconds=1\n\n// after: more attempts with exponential backoff\nfs.bos.input.stream.retry=6\nfs.bos.input.stream.retry.interval.seconds=2","handlingStrategy":"retry","validationCode":null,"typeGuard":"static boolean isRetryExhaustedRead(IOException e) {\n  return e.getMessage() != null && e.getMessage().startsWith(\"Retry \")\n      && e.getMessage().contains(\"still exception\");\n}","tryCatchPattern":"catch (IOException e) {\n  if (isRetryExhaustedRead(e) && e.getCause() != null) {\n    LOG.error(\"underlying failure\", e.getCause()); // diagnose the cause, not the wrapper\n  }\n  in = fs.open(path); // full reopen from scratch\n  in.seek(lastGoodPos); // resume from checkpoint\n}","preventionTips":["Checkpoint read positions so a failed stream can be reopened and resumed","Size the retry budget to your network's worst blip","Use credentials whose TTL exceeds the longest task"],"tags":["bos","read-failure","retry-exhausted","network","hadoop"],"backgroundTag":"retry-limit-exceeded","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}