apache/hadoop · error · EOFException

HTTP stream closed before all bytes were read. Expected %,d

Error message

HTTP stream closed before all bytes were read. Expected %,d bytes but only read %,d bytes. Current position %,d (%s)

What it means

Thrown while filling one requested range during readVectored(): objectContent.read() returned -1 before the range's length bytes arrived. The range was issued as a Content-Range GET, so a short body means the connection was cut or the object changed mid-transfer; S3A aborts that range with this EOFException.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java:1131

   * @throws EOFException if EOF if read() call returns -1
   * @throws InterruptedIOException if vectored IO operation is stopped.
   */
  private void readByteArray(InputStream objectContent,
                            final FileRange range,
                            byte[] dest,
                            int offset,
                            int length) throws IOException {
    LOG.debug("Reading {} bytes", length);
    int readBytes = 0;
    long position = range.getOffset();
    while (readBytes < length) {
      checkIfVectoredIOStopped();
      int readBytesCurr = objectContent.read(dest,
              offset + readBytes,
              length - readBytes);
      LOG.debug("read {} bytes from stream", readBytesCurr);
      if (readBytesCurr < 0) {
        throw new EOFException(
            String.format("HTTP stream closed before all bytes were read."
                    + " Expected %,d bytes but only read %,d bytes. Current position %,d"
                    + " (%s)",
                length, readBytes, position, range));
      }
      readBytes += readBytesCurr;
      position += readBytesCurr;

      // update io stats incrementally
      incrementBytesRead(readBytesCurr);
    }
  }

  /**
   * Read data from S3 with retries for the GET request
   * This also handles if file has been changed while the
   * http call is getting executed. If the file has been
   * changed RemoteFileChangedException is thrown.

View on GitHub (pinned to 2add963021)

Solutions

Retry the read by reopening the stream - a new GET re-issues the exact range
If persistent, check fs.s3a.connection.timeout, socket buffer settings, and any proxy idle-timeout on the S3 endpoint
Verify the object's current length/ETag when concurrent overwrite is possible and fail fast with a clearer error
As a workaround, use plain positioned reads (readFully) instead of vectored reads for the affected job

Example fix

// before
List<CompletableFuture<ByteBuffer>> futs = in.readVectored(ranges, ByteBuffer::allocate);
ByteBuffer b = futs.get(0).join(); // EOFException: HTTP stream closed before all bytes were read

// after - retry the whole vectored read on a fresh stream
for (int i = 1; i <= 3; i++) {
  try (FSDataInputStream fresh = fs.open(path)) {
    List<CompletableFuture<ByteBuffer>> f = fresh.readVectored(ranges, ByteBuffer::allocate);
    return f.get(0).join();
  } catch (CompletionException e) {
    if (!(e.getCause() instanceof EOFException) || i == 3) throw e;
  }
}

Defensive patterns

Strategy: retry

Validate before calling

long len = fs.getFileStatus(path).getLen();
for (ByteRange r : ranges) {
  if (r.getOffset() < 0 || r.getOffset() + r.getLength() > len) {
    throw new IllegalArgumentException("Range outside object: " + r);
  }
}

Try / catch

catch EOFException (or CompletionException/ExecutionException wrapping it from the futures), reopen the stream, and retry the same ranges a bounded number of times with backoff; verify the object length between attempts if concurrent rewrites are possible

Prevention

Set fs.s3a.connection.timeout appropriate to the size of vectored ranges
Fix proxy/NAT idle timeouts that cut long S3 GETs
Watch S3A instrumentation for vectored-read failures to detect flaky links early

When it happens

Trigger: readVectored() where a range's body terminates early: connection reset between the cluster and S3, S3 closing the socket mid-body, or the object being truncated/replaced while the range streams.

Common situations: Network instability or NAT/proxy timeouts during large columnar reads; objects actively rewritten by another writer during a long query; intermittent S3 connection drops on long-running scans.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8853cb4ed10d41de. Report an issue: GitHub.