{"record":{"id":"92264a3159246943","repo":"apache/hadoop","slug":"read-failed-of-inputstream-is","errorCode":null,"errorMessage":"read failed of {}, inputStream is {}","messagePattern":"read failed of (.+?), inputStream is (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSInputStream.java","lineNumber":1023,"sourceCode":"                position, length, offset, bytesRead, uri, retryTime,\n                exception, e);\n            throw exception;\n          }\n        }\n      } finally {\n        if (inputStream != null) {\n          inputStream.close();\n        }\n      }\n    }\n\n    if (inputStream == null || exception != null) {\n      LOG.error(\n          \"read position[{}] destLen[{}] offset[{}] len[{}] failed, \"\n              + \"retry time[{}], due to exception[{}]\",\n          position, length, offset, bytesRead, READ_RETRY_TIME,\n          exception);\n      throw new IOException(\"read failed of \" + uri + \", inputStream is \"\n          + (inputStream == null ? \"null\" : \"not null\"), exception);\n\n    }\n\n    long endTime = System.currentTimeMillis();\n    LOG.debug(\n        \"Read-4args uri:{}, contentLength:{}, destLen:{}, readLen:{}, \"\n            + \"position:{}, thread:{}, timeUsedMilliSec:{}\",\n        uri, contentLength, length, bytesRead, position, threadId,\n        endTime - startTime);\n    return bytesRead;\n  }\n\n  @Override\n  public synchronized void setReadahead(final Long newReadaheadRange) {\n    if (newReadaheadRange == null) {\n      this.readAheadRange = OBSConstants.DEFAULT_READAHEAD_RANGE;\n    } else {","sourceCodeStart":1005,"sourceCodeEnd":1041,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSInputStream.java#L1005-L1041","documentation":"OBSInputStream's positioned read (read(position, buffer, offset, length)) issues its own ranged GET with retries; after the retry loop, if the inputStream obtained is null or a caught exception is non-null, it logs the detailed failure and throws IOException('read failed of ' + uri + ', inputStream is ' + (null|'not null'), exception). This is the terminal failure of a positional read — the underlying OBS ranged GET could not be established or kept alive across READ_RETRY_TIME attempts, and the cause is attached for diagnosis.","triggerScenarios":"Positional/parallel reads (read(position, buf, off, len), e.g. from SplitLocationAware readers or the 'Read-4args' path) when every ranged GET attempt fails: network breakage to OBS, expired credentials mid-read, throttling (403/429), object deleted or overwritten mid-read (412/404), or proxy termination of range requests.","commonSituations":"Long-running jobs whose OBS credentials expire partway; connectivity blips between the cluster and the OBS endpoint; aggressive parallel readers exceeding QPS limits; objects replaced concurrently by writers so ranged reads fail consistency checks; misconfigured read-ahead vs. connection pool exhaustion causing null streams.","solutions":["Inspect the attached cause exception (and the preceding ERROR log line with retry count) — fix that root cause: refresh credentials, raise fs.obs.connection limits, or back off parallelism","Add caller-side bounded retry with exponential backoff for transient causes (timeout, 429, 5xx); do not retry deterministic 404/410","If objects are concurrently overwritten, switch readers to a consistent snapshot (copy/commit-then-read) instead of live positional reads","Tune fs.obs.readahead.range / thread pools if the error correlates with heavy positional-read fan-out"],"exampleFix":"// before\nint n = in.read(position, buf, offset, length); // no handling -> IOException propagates, job dies\n\n// after\nint readWithBackoff(FSDataInputStream in, long pos, byte[] buf, int off, int len) throws IOException {\n  long backoff = 200;\n  for (int i = 0; i < 4; i++) {\n    try {\n      return in.read(pos, buf, off, len);\n    } catch (IOException e) {\n      if (i == 3 || !isTransient(e)) throw e;\n      try { Thread.sleep(backoff); } catch (InterruptedException ie) {\n        Thread.currentThread().interrupt(); throw e;\n      }\n      backoff *= 2;\n    }\n  }\n  throw new IOException(\"unreachable\");\n}\nboolean isTransient(IOException e) {\n  Throwable c = e.getCause();\n  String m = c == null ? \"\" : String.valueOf(c.getMessage());\n  return m.contains(\"Timeout\") || m.contains(\"429\") || m.contains(\"50\");\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  return in.read(position, buffer, offset, length);\n} catch (IOException e) {\n  Throwable cause = e.getCause();\n  String msg = cause == null ? \"\" : String.valueOf(cause.getMessage());\n  boolean transient_ = msg.contains(\"Timeout\") || msg.contains(\"429\") || msg.contains(\"503\");\n  if (transient_) {\n    return retryWithBackoff(in, position, buffer, offset, length, 3, 200);\n  }\n  // deterministic (404/410/permission): surface cause for diagnosis\n  throw e;\n}","preventionTips":["Read the attached cause before retrying — 404/410/permission failures are not retryable","Keep credentials fresh for long jobs; schedule refresh before expiry","Cap parallel positional readers within OBS QPS limits and configure connection pools adequately"],"tags":["network","read","retry-exhausted","hadoop-obs"],"backgroundTag":"object-storage-read-failure","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}