apache/hadoop · error · EOFException

End of stream reached draining data between ranges; expected

Error message

End of stream reached draining data between ranges; expected %,d bytes; only drained %,d bytes before -1 returned (position=%,d)

What it means

Thrown during readVectored() on S3AInputStream. When requested ranges are close enough to be served from one HTTP stream, S3A drains (reads and discards) the gap between them; this EOFException means the S3 stream returned -1 before drainQuantity gap bytes were consumed. The source comment states no recovery is attempted and network issues are the usual cause.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java:1012

    byte[] drainBuffer;
    int size = (int)Math.min(InternalConstants.DRAIN_BUFFER_SIZE, drainQuantity);
    drainBuffer = new byte[size];
    LOG.debug("Draining {} bytes from stream from offset {}; buffer size={}",
        drainQuantity, position, size);
    try {
      long remaining = drainQuantity;
      while (remaining > 0) {
        checkIfVectoredIOStopped();
        readCount = objectContent.read(drainBuffer, 0, (int)Math.min(size, remaining));
        LOG.debug("Drained {} bytes from stream", readCount);
        if (readCount < 0) {
          // read request failed; often network issues.
          // no attempt is made to recover at this point.
          final String s = String.format(
              "End of stream reached draining data between ranges; expected %,d bytes;"
                  + " only drained %,d bytes before -1 returned (position=%,d)",
              drainQuantity, drainBytes, position + drainBytes);
          throw new EOFException(s);
        }
        drainBytes += readCount;
        remaining -= readCount;
      }
    } finally {
      getS3AStreamStatistics().readVectoredBytesDiscarded(drainBytes);
      LOG.debug("{} bytes drained from stream ", drainBytes);
    }
  }

  /**
   * Read data from S3 for this range and populate the buffer.
   * @param range range of data to read.
   * @param buffer buffer to fill.
   */
  private void readSingleRange(FileRange range, ByteBuffer buffer) {
    LOG.debug("Start reading {} from {} ", range, getPathStr());
    if (range.getLength() == 0) {

View on GitHub (pinned to 2add963021)

Solutions

1. Retry the readVectored operation on a fresh stream - this failure is not auto-recovered inside S3A
2. Tune fs.s3a.connection.timeout / keepalive settings and check proxy/NAT idle timeouts on the S3 network path
3. Reduce gap draining: lower fs.s3a.vectored.read.max.merged.size or raise fs.s3a.vectored.read.min.seek.size so distant ranges use separate requests instead of one long drain
4. If it recurs on the same object/ranges, verify the object length and that nothing is concurrently rewriting it

Example fix

// before
try (FSDataInputStream in = fs.open(path)) {
  List<CompletableFuture<ByteBuffer>> f = in.readVectored(ranges, ByteBuffer::allocate);
  f.forEach(cf -> cf.join()); // EOF while draining gap between ranges
}

// after - retry with backoff on a fresh stream
for (int attempt = 1; attempt <= 3; attempt++) {
  try (FSDataInputStream in = fs.open(path)) {
    in.readVectored(ranges, ByteBuffer::allocate).forEach(cf -> cf.join());
    break;
  } catch (CompletionException e) {
    if (!(e.getCause() instanceof EOFException) || attempt == 3) throw e;
    Thread.sleep(100L << attempt);
  }
}

long len = fs.getFileStatus(path).getLen();
for (ByteRange r : ranges) {
  if (r.getOffset() < 0 || r.getOffset() + r.getLength() > len) {
    throw new IllegalArgumentException("Range outside object: " + r);
  }
}

catch the EOFException surfacing from the vectored-read futures (often wrapped in CompletionException/ExecutionException), reopen the stream with fs.open(path), and retry the readVectored with capped exponential backoff; give up after a few attempts and surface the original error

When it happens

Trigger: readVectored(List<ByteRange>, ...) where ranges get merged into one request and the inter-range gap must be drained, and the HTTP response body is cut short by a connection reset, proxy/NAT idle timeout, or the object being replaced by a shorter one mid-read.

Common situations: Vectored (columnar) reads from ORC/Hive/Spark over flaky links; corporate proxies or load balancers terminating long S3 GETs; NAT idle timeouts on slow transfers; S3 transient connection drops.

Related errors

• HTTP stream closed before all bytes were read. Expected %,d
• Cannot seek to a negative offset <targetPos>
• End of file reached before reading fully.
• Stream closed or unbuffer is called
• getCredentials failed: {e}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/fef527ef098c15aa. Report an issue: GitHub.