apache/hadoop · error · IOException

Unexpected end of stream during direct read: path={path}, of

Error message

Unexpected end of stream during direct read: path={path}, offset={offset}, requested={requested}

What it means

Error "Unexpected end of stream during direct read: path={path}, offset={offset}, requested={requested}" thrown in apache/hadoop.

Source

Thrown at hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/VectoredReadHandler.java:570

      AbfsInputStream stream,
      CombinedFileRange unit,
      IntFunction<ByteBuffer> allocator) throws IOException {

    LOG.debug("directRead: path={}, offset={}, length={}",
        stream.getPath(), unit.getOffset(), unit.getLength());

    /* Read entire combined range into a temporary buffer */
    byte[] tmp = new byte[unit.getLength()];
    TracingContext tracingContext = new TracingContext(stream.getTracingContext());
    tracingContext.setReadType(ReadType.VECTORED_DIRECT_READ);

    int total = 0;
    int requested = unit.getLength();
    while (total < requested) {
      int n = stream.readRemote(unit.getOffset() + total, tmp, total,
          requested - total, tracingContext);
      if (n <= 0) {
        throw new IOException(
            "Unexpected end of stream during direct read: path=" + stream.getPath()
                + ", offset=" + (unit.getOffset() + total)
                + ", requested=" + requested);
      }
      total += n;
    }

    LOG.debug("directRead: read complete: path={}, offset={}, bytesRead={}",
        stream.getPath(), unit.getOffset(), total);

    long unitStart = unit.getOffset();
    long unitEnd = unitStart + unit.getLength();

    /* Distribute data to each logical FileRange */
    for (FileRange r : unit.getUnderlying()) {
      CompletableFuture<ByteBuffer> future = r.getData();
      if (future == null || future.isDone()) {
        continue;

View on GitHub (pinned to 2add963021)

Solutions

1. Retry the read; premature stream end usually indicates a truncated service response or network failure.
2. If persistent, capture logs for the path/offset and report to the service.

When it happens

Trigger: A direct vectored read receives fewer bytes than requested before the stream ends.

Common situations: See trigger scenarios.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/da40c100f8182e7d. Report an issue: GitHub.