apache/hadoop · error · IOException

Null IO stream from reopen of (" + reason + ") " + key

Error message

Null IO stream from reopen of (" + reason + ") " + key

What it means

During reopen (lazy seek or read-failure recovery) BosInputStream calls store.retrieve(key, targetPos, contentRangeFinish, ...). If the call either returns null or throws — the catch block closes the stream, leaving in == null — it throws IOException 'Null IO stream from reopen of (<reason>) <key>'. The true cause is not in this exception; it appears only in the preceding LOG.info 'native store retrieve failed reason : ...' line.

Source

Thrown at hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BosInputStream.java:559

    LOG.info(
        "reopen({}) for {} range[{}-{}], length={},"
            + " contentLength={}, streamPosition={},"
            + " nextReadPosition={}",
        key, reason, targetPos, contentRangeFinish,
        length, contentLength, pos, nextReadPos);
    try {
      in = store.retrieve(
          key, targetPos, contentRangeFinish,
          fileMetaData);
      contentRangeStart = targetPos;
    } catch (Exception e) {
      LOG.info(
          "native store retrieve failed reason : {}",
          e.getMessage());
      closeStream("native store retrieve failed");
    }
    if (in == null) {
      throw new IOException(
          "Null IO stream from reopen of ("
              + reason + ") " + key);
    }
    this.pos = targetPos;
  }

  /**
   * Calculate the limit for a get request.
   *
   * @param targetPos position of the read
   * @param length length of bytes requested; if less than
   *               zero "unknown"
   * @param fileContentLength total length of file
   * @param readaheadLen current readahead value
   * @return the absolute value of the limit of the request
   *         (inclusive, last byte index).
   */
  private long calculateRequestLimit(

View on GitHub (pinned to 2add963021)

Solutions

1. Correlate with the server log line 'native store retrieve failed reason' — it carries the actual underlying error
2. Recover by reopening the file from scratch (new open() + seek to last good position) rather than reusing the dead stream
3. Verify the object still exists and its length is stable between open and read; forbid in-place rewrites of keys being read
4. Refresh long-lived credentials / re-issue STS tokens with longer TTL for long jobs

static boolean isReopenFailure(IOException e) {
  return e.getMessage() != null && e.getMessage().startsWith("Null IO stream from reopen");
}

catch (IOException e) {
  if (isReopenFailure(e)) {
    in = fs.open(path);            // full reopen
    ((BosInputStream) in.getWrappedStream()).seek(lastGoodPos);
  } else { throw e; }
}

When it happens

Trigger: The recovery GET for a byte range fails or yields no content: object deleted or truncated mid-read, range no longer valid, auth/session-token expiry mid-stream, or a transient BOS 5xx during the reopen itself.

Common situations: Objects overwritten/deleted while a long task reads them; expired STS credentials during extended jobs; reading a file whose writer has not committed all data yet; flaky network to the BOS endpoint.

Related errors

• Retry " + retry + " times to read still exception: " + error
• RequestRateLimitExceeded
• status code 429 !!!" + e.getCause()
• Invalid read parameters: buf.length=%d, off=%d, len=%d
• Thread interrupted during retry

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9546a920f58d6362. Report an issue: GitHub.