apache/hadoop · error · EOFException

Unexpected end of stream: buffer[%d], readSize = %d, numRema

Error message

Unexpected end of stream: buffer[%d], readSize = %d, numRemainingBytes = %d

What it means

S3ARemoteObjectReader.readToBuffer() reads exactly the remaining bytes of a fetched block from the remote/block stream; if inputStream.read() returns -1 before numRemainingBytes reaches 0, it throws EOFException("Unexpected end of stream: buffer[...], readSize = ..., numRemainingBytes = ..."). The S3 (or cached) stream ended earlier than the content length promised - a truncated read, not normal EOF.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/prefetch/S3ARemoteObjectReader.java:165

    }

    ResponseInputStream<GetObjectResponse> inputStream =
        remoteObject.openForRead(offset, readSize);
    int numRemainingBytes = readSize;
    byte[] bytes = new byte[READ_BUFFER_SIZE];

    int numBytesToRead;
    int numBytes;

    try {
      do {
        numBytesToRead = Math.min(READ_BUFFER_SIZE, numRemainingBytes);
        numBytes = inputStream.read(bytes, 0, numBytesToRead);
        if (numBytes < 0) {
          String message = String.format(
              "Unexpected end of stream: buffer[%d], readSize = %d, numRemainingBytes = %d",
              buffer.capacity(), readSize, numRemainingBytes);
          throw new EOFException(message);
        }

        if (numBytes > 0) {
          buffer.put(bytes, 0, numBytes);
          numRemainingBytes -= numBytes;
        }
      }
      while (!this.closed && (numRemainingBytes > 0));
    } finally {
      remoteObject.close(inputStream, numRemainingBytes);
    }
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. Retry the failing operation - truncation from network resets is typically transient and S3A read retries usually resolve it; application-level task retry catches the rest.
2. Check disk space and health of the prefetch/block cache directory; clean or reconfigure (buffer dir location/size) if it is being truncated or evicted mid-read.
3. Review proxy/load-balancer idle timeouts on the path to S3 (keep-alive settings) if resets recur at consistent intervals.
4. If it persists on the same object/offset, verify the object was not replaced/deleted mid-read and that no external process touches the cache dir.

Example fix

// application-level guard (e.g. in a Spark task or job runner)
try {
  return readThrough(stream);
} catch (EOFException e) {
  // truncated read of prefetched block: retry the whole operation
  if (attempt < MAX_RETRIES) {
    stream.close();
    stream = fs.open(path); // fresh stream, fresh block fetch
    return readThrough(stream);
  }
  throw e;
}

for (int attempt = 1; attempt <= 3; attempt++) {
  try (FSDataInputStream in = fs.open(path)) {
    return readBlock(in, offset, len);
  } catch (EOFException e) {
    if (attempt == 3) throw e;
    // truncated transfer: next attempt reopens and refetches
  }
}

When it happens

Trigger: Network connection to S3 cut mid-block (proxy idle timeout, connection reset); block cache data file shorter than expected (disk full, partial write, external deletion of cache dir contents); object deleted or truncated concurrently; retry policy gave up partway through a ranged GET.

Common situations: Long-running Spark/Hive jobs on flaky networks or aggressive proxies; nodes whose prefetch cache disk fills; S3 request throttling mid-transfer; antivirus/janitor processes cleaning the prefetch buffer directory while a job runs.

Related errors

• Attempted to seek or read past the end of the file " + pos
• Number of parts in multipart upload exceeded. Current part c
• Stream is closed!
• name + ": Stream is closed!"
• Cannot seek to a negative offset " + pos

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f2737e42a9420ba2. Report an issue: GitHub.