apache/hadoop · error · EOFException

End of file reached before reading fully.

Error message

End of file reached before reading fully.

What it means

Thrown from S3AInputStream.readFully(position, buffer, offset, length) when the underlying read() returns -1 before length bytes were transferred. The original position is restored via seekQuietly in the finally block, but the call still fails with FSExceptionMessages.EOF_IN_READ_FULLY: the object ended before the requested range was satisfied.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java:819

      throws IOException {
    checkNotClosed();
    validatePositionedReadArgs(position, buffer, offset, length);
    getS3AStreamStatistics().readFullyOperationStarted(position, length);
    if (length == 0) {
      return;
    }
    int nread = 0;
    synchronized (this) {
      long oldPos = getPos();
      try {
        seek(position);
        while (nread < length) {
          int nbytes = read(buffer, offset + nread, length - nread);
          if (nbytes < 0) {
            // no attempt is currently made to recover from stream read problems;
            // a lazy seek to the offset is probably the solution.
            // but it will need more qualification against failure handling
            throw new EOFException(FSExceptionMessages.EOF_IN_READ_FULLY);
          }
          nread += nbytes;
        }
      } finally {
        seekQuietly(oldPos);
      }
    }
  }

  /**
   * {@inheritDoc}
   * Pass to {@link #readVectored(List, IntFunction, Consumer)}
   * with the {@link VectoredReadUtils#LOG_BYTE_BUFFER_RELEASED} releaser.
   * @param ranges the byte ranges to read.
   * @param allocate the function to allocate ByteBuffer.
   * @throws IOException IOE if any.
   */
  @Override

View on GitHub (pinned to 2add963021)

Solutions

1. Check position + length against fs.getFileStatus(path).getLen() before readFully and treat short files as invalid/empty input
2. Catch EOFException and handle 'file too short' explicitly with a clearer domain error instead of letting it propagate raw
3. If concurrent writes are possible, verify the object was not truncated/overwritten between status and read
4. For footer formats, validate the magic/size fields against the actual file length before issuing the read

Example fix

// before
in.readFully(pos, buf, 0, FOOTER_LEN); // EOF: End of file reached before reading fully

// after
long remaining = fileStatus.getLen() - pos;
if (remaining < FOOTER_LEN) {
  throw new IOException("File " + path + " too short: need " + FOOTER_LEN
      + " bytes at " + pos + ", only " + remaining + " available");
}
in.readFully(pos, buf, 0, FOOTER_LEN);

long remaining = fs.getFileStatus(path).getLen() - position;
if (remaining < length) {
  throw new IOException("Requested " + length + " bytes at " + position
      + " but only " + Math.max(remaining, 0) + " remain in " + path);
}
in.readFully(position, buffer, offset, length);

try { in.readFully(pos, buf, off, len); } catch (EOFException e) { /* file is shorter than assumed - fail with a domain error or retry with a smaller read; do not blindly retry the same length */ }

When it happens

Trigger: readFully(position, buf, off, len) where position + len exceeds the object's length; reading a fixed-size header/footer from a file shorter than assumed; reading an object that was concurrently truncated or replaced with a smaller one.

Common situations: Columnar readers (ORC/Parquet/Avro) reading footers from tiny, empty, or truncated files; zero-byte uploads; ranges computed from a stale FileStatus while the object changed; split logic assuming a minimum file size.

Related errors

• Cannot seek to a negative offset <targetPos>
• End of stream reached draining data between ranges; expected
• HTTP stream closed before all bytes were read. Expected %,d
• <uri>: Stream is closed!
• Stream closed or unbuffer is called

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/096768b88f071b1a. Report an issue: GitHub.