apache/hadoop · error · EOFException

End of file reached before reading fully.

Error message

End of file reached before reading fully.

What it means

DFSInputStream.readFully(long position, ByteBuffer buf) loops read(position + nread, buf) until the buffer is full; if a positional read returns -1 (EOF) before the buffer fills, it throws EOFException('End of file reached before reading fully.'). readFully is all-or-nothing by contract: the requested range must lie entirely within [0, fileLength). The position argument is absolute, not relative to the current stream position.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java:1774

    throw new IOException("Mark/reset not supported");
  }

  @Override
  public int read(long position, final ByteBuffer buf) throws IOException {
    if (!buf.hasRemaining()) {
      return 0;
    }
    return pread(position, buf);
  }

  @Override
  public void readFully(long position, final ByteBuffer buf)
      throws IOException {
    int nread = 0;
    while (buf.hasRemaining()) {
      int nbytes = read(position + nread, buf);
      if (nbytes < 0) {
        throw new EOFException(FSExceptionMessages.EOF_IN_READ_FULLY);
      }
      nread += nbytes;
    }
  }

  /** Utility class to encapsulate data node info and its address. */
  static final class DNAddrPair {
    final DatanodeInfo info;
    final InetSocketAddress addr;
    final StorageType storageType;
    final LocatedBlock block;

    DNAddrPair(DatanodeInfo info, InetSocketAddress addr,
        StorageType storageType, LocatedBlock block) {
      this.info = info;
      this.addr = addr;
      this.storageType = storageType;
      this.block = block;

View on GitHub (pinned to 2add963021)

Solutions

1. Validate up front: long remaining = fs.getFileStatus(path).getLen() - position; ensure remaining >= buf.remaining() before calling readFully.
2. If the file may be changing underneath you, re-stat the length on EOFException and decide whether to retry (file grew) or fail (file truncated).
3. If partial data is acceptable, replace readFully with a positional read() loop that tolerates -1 instead of demanding the full range.
4. If you control the writer, make truncation/rewrite atomic (write to temp path then rename) so readers never see a shrinking file.

Example fix

// before
ByteBuffer footer = ByteBuffer.allocate(FOOTER_LEN);
in.readFully(fileLenCached - FOOTER_LEN, footer); // throws if file shrank

// after
long fileLen = fs.getFileStatus(path).getLen(); // fresh length
ByteBuffer footer = ByteBuffer.allocate(FOOTER_LEN);
if (fileLen < FOOTER_LEN) throw new EOFException(path + " too small for footer");
in.readFully(fileLen - FOOTER_LEN, footer);

long fileLen = fs.getFileStatus(path).getLen(); // fresh, not cached
long remaining = fileLen - position;
if (remaining < 0) throw new IllegalArgumentException("position past EOF: " + position);
if (buf.remaining() > remaining) buf.limit(buf.position() + (int) remaining);
in.readFully(position, buf);

try {
  in.readFully(position, buf);
} catch (EOFException e) {
  long freshLen = fs.getFileStatus(path).getLen(); // file may have been truncated
  if (position + nread < freshLen) retryReadFully(position + nread, buf); // grew: continue
  else throw new IllegalStateException("file truncated under reader: " + path, e);
}

When it happens

Trigger: readFully(pos, buf) where pos + buf.remaining() > file length; offsets computed from a stale file length obtained before another process truncated the file; racing append (reader stat'ed length L, file truncated to less before the read).

Common situations: Reading fixed-size footers/trailers at (fileLen - footerSize) when the file is shorter than footerSize; MapReduce/Spark input formats computing split boundaries against a cached file length; concurrent compaction or rewrite jobs truncating files under active readers.

Related errors

• Attempted to read past end of file
• Cannot seek after EOF
• {b}'s on-disk length {onDiskLength} is shorter than minLengt
• Reached EOF when reading log header
• Could not find target position {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c1936f9f6aa94bd6. Report an issue: GitHub.