apache/hadoop · error · IOException

read failed of {}, inputStream is {}

Error message

read failed of {}, inputStream is {}

What it means

OBSInputStream's positioned read (read(position, buffer, offset, length)) issues its own ranged GET with retries; after the retry loop, if the inputStream obtained is null or a caught exception is non-null, it logs the detailed failure and throws IOException('read failed of ' + uri + ', inputStream is ' + (null|'not null'), exception). This is the terminal failure of a positional read — the underlying OBS ranged GET could not be established or kept alive across READ_RETRY_TIME attempts, and the cause is attached for diagnosis.

Source

Thrown at hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSInputStream.java:1023

                position, length, offset, bytesRead, uri, retryTime,
                exception, e);
            throw exception;
          }
        }
      } finally {
        if (inputStream != null) {
          inputStream.close();
        }
      }
    }

    if (inputStream == null || exception != null) {
      LOG.error(
          "read position[{}] destLen[{}] offset[{}] len[{}] failed, "
              + "retry time[{}], due to exception[{}]",
          position, length, offset, bytesRead, READ_RETRY_TIME,
          exception);
      throw new IOException("read failed of " + uri + ", inputStream is "
          + (inputStream == null ? "null" : "not null"), exception);

    }

    long endTime = System.currentTimeMillis();
    LOG.debug(
        "Read-4args uri:{}, contentLength:{}, destLen:{}, readLen:{}, "
            + "position:{}, thread:{}, timeUsedMilliSec:{}",
        uri, contentLength, length, bytesRead, position, threadId,
        endTime - startTime);
    return bytesRead;
  }

  @Override
  public synchronized void setReadahead(final Long newReadaheadRange) {
    if (newReadaheadRange == null) {
      this.readAheadRange = OBSConstants.DEFAULT_READAHEAD_RANGE;
    } else {

View on GitHub (pinned to 2add963021)

Solutions

1. Inspect the attached cause exception (and the preceding ERROR log line with retry count) — fix that root cause: refresh credentials, raise fs.obs.connection limits, or back off parallelism
2. Add caller-side bounded retry with exponential backoff for transient causes (timeout, 429, 5xx); do not retry deterministic 404/410
3. If objects are concurrently overwritten, switch readers to a consistent snapshot (copy/commit-then-read) instead of live positional reads
4. Tune fs.obs.readahead.range / thread pools if the error correlates with heavy positional-read fan-out

Example fix

// before
int n = in.read(position, buf, offset, length); // no handling -> IOException propagates, job dies

// after
int readWithBackoff(FSDataInputStream in, long pos, byte[] buf, int off, int len) throws IOException {
  long backoff = 200;
  for (int i = 0; i < 4; i++) {
    try {
      return in.read(pos, buf, off, len);
    } catch (IOException e) {
      if (i == 3 || !isTransient(e)) throw e;
      try { Thread.sleep(backoff); } catch (InterruptedException ie) {
        Thread.currentThread().interrupt(); throw e;
      }
      backoff *= 2;
    }
  }
  throw new IOException("unreachable");
}
boolean isTransient(IOException e) {
  Throwable c = e.getCause();
  String m = c == null ? "" : String.valueOf(c.getMessage());
  return m.contains("Timeout") || m.contains("429") || m.contains("50");
}

try {
  return in.read(position, buffer, offset, length);
} catch (IOException e) {
  Throwable cause = e.getCause();
  String msg = cause == null ? "" : String.valueOf(cause.getMessage());
  boolean transient_ = msg.contains("Timeout") || msg.contains("429") || msg.contains("503");
  if (transient_) {
    return retryWithBackoff(in, position, buffer, offset, length, 3, 200);
  }
  // deterministic (404/410/permission): surface cause for diagnosis
  throw e;
}

When it happens

Trigger: Positional/parallel reads (read(position, buf, off, len), e.g. from SplitLocationAware readers or the 'Read-4args' path) when every ranged GET attempt fails: network breakage to OBS, expired credentials mid-read, throttling (403/429), object deleted or overwritten mid-read (412/404), or proxy termination of range requests.

Common situations: Long-running jobs whose OBS credentials expire partway; connectivity blips between the cluster and the OBS endpoint; aggressive parallel readers exceeding QPS limits; objects replaced concurrently by writers so ranged reads fail consistency checks; misconfigured read-ahead vs. connection pool exhaustion causing null streams.

Related errors

• Null IO stream from reopen of ({}) {}
• Retry " + retry + " times to read still exception: " + error
• Received end of stream result before all requestedBytes were
• End of file reached before reading fully.
• Exception occurred while closing channel '%s'

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/92264a3159246943. Report an issue: GitHub.