apache/hadoop · error · EOFException

End of file reached before reading fully.

Error message

End of file reached before reading fully.

What it means

OBSInputStream.readFully(position, buffer, offset, length) seeks to the requested position and loops read() until exactly length bytes are delivered; if read() returns -1 while nread < length it throws EOFException(FSExceptionMessages.EOF_IN_READ_FULLY, 'End of file reached before reading fully.'). readFully's contract is all-or-nothing: unlike read(), a short result at EOF is an error. The original position is restored in a finally via seekQuietly, so the stream stays usable after the exception.

Source

Thrown at hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSInputStream.java:906

      final int offset,
      final int length)
      throws IOException {
    long startTime = System.currentTimeMillis();
    long threadId = Thread.currentThread().getId();
    checkNotClosed();
    validatePositionedReadArgs(position, buffer, offset, length);
    if (length == 0) {
      return;
    }
    int nread = 0;
    synchronized (this) {
      long oldPos = getPos();
      try {
        seek(position);
        while (nread < length) {
          int nbytes = read(buffer, offset + nread, length - nread);
          if (nbytes < 0) {
            throw new EOFException(
                FSExceptionMessages.EOF_IN_READ_FULLY);
          }
          nread += nbytes;
        }
      } finally {
        seekQuietly(oldPos);
      }
    }

    long endTime = System.currentTimeMillis();
    LOG.debug(
        "ReadFully uri:{}, contentLength:{}, destLen:{}, readLen:{}, "
            + "position:{}, thread:{}, timeUsedMilliSec:{}",
        uri, contentLength, length, nread, position, threadId,
        endTime - startTime);
  }

  /**

View on GitHub (pinned to 2add963021)

Solutions

1. Clamp the request to the actual size: len = Math.min(len, fileStatus.getLen() - pos), and skip when len <= 0
2. Re-stat the file (getFileStatus) immediately before readFully when concurrent overwrites are possible
3. Treat this EOFException as a data condition: the file is shorter than the format promises — verify producer-side upload completion (commit markers) before consuming
4. For optional footers, fall back to a bounded read (loop read()) that tolerates EOF instead of readFully

Example fix

// before
in.readFully(pos, buf, 0, FOOTER_LEN); // file shorter than pos+FOER_LEN -> EOFException

// after
FileStatus st = fs.getFileStatus(path);
int len = (int) Math.min(FOOTER_LEN, st.getLen() - pos);
if (len < FOOTER_LEN) {
  throw new IOException("file " + path + " too small: " + st.getLen());
}
in.readFully(pos, buf, 0, len);

FileStatus st = fs.getFileStatus(path);
long avail = st.getLen() - position;
if (length > avail) {
  throw new EOFException("file " + path + " has only " + avail
      + " bytes at position " + position + "; requested " + length);
}
in.readFully(position, buffer, offset, length);

try {
  in.readFully(position, buffer, offset, length);
} catch (EOFException e) {
  // file shorter than format expects: fail with context, keep original position (stream restores it)
  throw new IOException("truncated object at " + path + ": " + e.getMessage(), e);
}

When it happens

Trigger: readFully(pos, buf, off, len) where pos+len exceeds the object's contentLength (e.g. reading a fixed-size footer/header from a truncated file); stale FileStatus length cached before the object was overwritten with a shorter one; computing reads from file size minus a constant when the file is smaller than that constant.

Common situations: Reading fixed-size file formats (magic bytes, footers) from files that are shorter than expected due to interrupted uploads; racing overwrites where another writer replaced the object mid-read; split computations using outdated lengths; zero-byte objects read with length>0.

Related errors

• Attempted to seek or read past the end of the file
• Null IO stream from reopen of ({}) {}
• read failed of {}, inputStream is {}
• Could not find target position {}
• Attempted to read past end of file

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b849a3a48b43d6f6. Report an issue: GitHub.