apache/hadoop · error · EOFException

Attempted to seek or read past the end of the file " + pos

Error message

Attempted to seek or read past the end of the file " + pos

What it means

S3ARemoteInputStream.throwIfInvalidSeek() throws EOFException(FSExceptionMessages.CANNOT_SEEK_PAST_EOF + " " + pos) when the seek target exceeds the size of the block data it is serving (pos > getBlockData().getFileSize()). Because this stream reads prefetcher block/cache data, the mismatch means either a genuine seek past the object end or a shrunken/corrupt local block file.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/prefetch/S3ARemoteInputStream.java:465

    StringBuilder sb = new StringBuilder();
    sb.append(String.format("nextReadPos = (%d)%n", nextReadPos));
    sb.append(String.format("fpos = (%s)", fpos));
    return sb.toString();
  }

  protected void throwIfClosed() throws IOException {
    if (closed) {
      throw new IOException(
          name + ": " + FSExceptionMessages.STREAM_IS_CLOSED);
    }
  }

  protected void throwIfInvalidSeek(long pos) throws EOFException {
    if (pos < 0) {
      throw new EOFException(FSExceptionMessages.NEGATIVE_SEEK + " " + pos);
    } else if (pos > this.getBlockData().getFileSize()) {
      throw new EOFException(FSExceptionMessages.CANNOT_SEEK_PAST_EOF + " " + pos);
    }
  }

  // Unsupported functions.

  @Override
  public void mark(int readlimit) {
    throw new UnsupportedOperationException("mark not supported");
  }

  @Override
  public void reset() {
    throw new UnsupportedOperationException("reset not supported");
  }

  @Override
  public long skip(long n) {
    throw new UnsupportedOperationException("skip not supported");

View on GitHub (pinned to 2add963021)

Solutions

Validate the target against the authoritative length: if (pos >= fs.getFileStatus(path).getLen()) stop or error before seeking.
If you suspect cache corruption, clear the prefetch/block cache directory and reopen the stream.
Ensure the cache directory has disk space and is not aggressively cleaned by external tools.
For objects that change underneath, reopen the stream (fresher metadata) rather than seeking far ahead on an old handle.

Example fix

// before
long len = status.getLen();
in.seek(len); // one past the last byte -> EOFException

// after
if (pos < fs.getFileStatus(path).getLen()) {
  in.seek(pos);
} else {
  // end of file reached
}

Defensive patterns

Strategy: validation

Validate before calling

long len = fs.getFileStatus(path).getLen();
if (pos >= len) {
  // genuine end of file: do not seek
  return -1;
}
in.seek(pos);

Try / catch

try {
  in.seek(pos);
} catch (EOFException e) {
  if (pos >= fs.getFileStatus(path).getLen()) {
    // true EOF: handle as end-of-input
  } else {
    // block cache shrank/corrupt: evict cache dir and reopen
  }
}

Prevention

Always compare seek targets against a fresh getFileStatus().getLen().
Use pos < len (strictly less) for the last valid byte.
Monitor disk space on the prefetch cache directory.

When it happens

Trigger: seek(pos) where pos >= object length (e.g. seeking to fileLen instead of fileLen-1); prefetch block data file truncated on disk (cache dir out of space, cleanup deleted it); object replaced by a shorter one mid-read and cached metadata is stale.

Common situations: Apps computing end-of-file positions with <= vs < errors; disk-full nodes truncating files under fs.s3a.prefetch.buffer.dir / block cache; long-lived readers on objects that were overwritten concurrently.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/da15c8eed7686b73. Report an issue: GitHub.