What causes "Attempted to seek or read past the end of the file" in apache/hadoop?

Bound the skip by remaining bytes: skip only min(requested, stream.size() - stream.position())

apache/hadoop · error · EOFException

Attempted to seek or read past the end of the file

Error message

Attempted to seek or read past the end of the file

What it means

The second positional check in ByteBufferInputStream.skip(): if position() + offset exceeds the block's fixed size, the method throws EOFException(FSExceptionMessages.CANNOT_SEEK_PAST_EOF). It fires when a skip request tries to move the read cursor beyond the end of the buffered block data. Like the negative variant, it is a pure client-side bounds check on the in-memory bytebuffer block — the stream never consults OBS.

Source

Thrown at hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSDataBlocks.java:716

      public synchronized int read() {
        if (available() > 0) {
          return byteBuffer.get() & OBSCommonUtils.BYTE_TO_INT_MASK;
        } else {
          return -1;
        }
      }

      @Override
      public synchronized long skip(final long offset)
          throws IOException {
        verifyOpen();
        long newPos = position() + offset;
        if (newPos < 0) {
          throw new EOFException(FSExceptionMessages.NEGATIVE_SEEK);
        }
        if (newPos > size) {
          throw new EOFException(
              FSExceptionMessages.CANNOT_SEEK_PAST_EOF);
        }
        byteBuffer.position((int) newPos);
        return newPos;
      }

      @Override
      public synchronized int available() {
        Preconditions.checkState(byteBuffer != null,
            FSExceptionMessages.STREAM_IS_CLOSED);
        return byteBuffer.remaining();
      }

      /**
       * Get the current buffer position.
       *
       * @return the buffer position
       */

View on GitHub (pinned to 2add963021)

Solutions

Bound the skip by remaining bytes: skip only min(requested, stream.size() - stream.position())
Treat EOFException from skip as a signal that the block is exhausted and exit the read loop gracefully
Prefer read()-based skipping (loop reading into a small buffer) when the exact remaining length is uncertain

Example fix

// before
blockStream.skip(requested); // throws if it passes the end of the block

// after
long canSkip = Math.min(requested, blockStream.size() - blockStream.position());
if (canSkip > 0) {
  blockStream.skip(canSkip);
}

Defensive patterns

Strategy: validation

Validate before calling

long remaining = blockStream.size() - blockStream.position();
long toSkip = Math.min(requested, remaining);
if (toSkip > 0) {
  blockStream.skip(toSkip);
}

Try / catch

try {
  blockStream.skip(requested);
} catch (EOFException e) {
  if (FSExceptionMessages.CANNOT_SEEK_PAST_EOF.equals(e.getMessage())) {
    // treat as end of block: consume the rest and signal EOF to the caller
    blockStream.skip(blockStream.size() - blockStream.position());
  } else {
    throw e;
  }
}

Prevention

Always bound skips by size() - position()
Prefer read-based skipping when remaining length is unknown
Handle EOFException from skip as a normal end-of-data signal in reader loops

When it happens

Trigger: Calling skip(n) with n larger than remaining bytes in the block (position + n > size); seeking forward past a small final block during upload buffering; a reader loop that assumes skip() silently stops at EOF and instead lets the exception propagate.

Common situations: Record readers that skip fixed-size headers/records against short blocks; code ported from FSDataInputStream.skip() semantics (which clamps at EOF) to raw block streams; arithmetic bugs where remaining length is computed against the wrong total.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6087190e33cdbbe7. Report an issue: GitHub.