apache/hadoop · error · IOException

Attempted to read past end of file

Error message

Attempted to read past end of file

What it means

DFSStripedInputStream.blockSeekTo(long target) - called by read paths to (re)position onto a block group of an erasure-coded (EC) file - throws IOException('Attempted to read past end of file') when target >= getFileLength(). For striped files this check sits at the top of the seek-to-block path, so any read/skip/seek continuation at or beyond EOF of an EC file surfaces here rather than as the replicated stream's EOFException variants.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java:160

  protected ByteBuffer getCurStripeBuf() {
    return curStripeBuf;
  }

  protected ByteBufferPool getBufferPool() {
    return BUFFER_POOL;
  }

  protected ThreadPoolExecutor getStripedReadsThreadPool(){
    return dfsClient.getStripedReadsThreadPool();
  }
  /**
   * When seeking into a new block group, create blockReader for each internal
   * block in the group.
   */
  @VisibleForTesting
  synchronized void blockSeekTo(long target) throws IOException {
    if (target >= getFileLength()) {
      throw new IOException("Attempted to read past end of file");
    }

    maybeRegisterBlockRefresh();

    // Will be getting a new BlockReader.
    closeCurrentBlockReaders();

    // Compute desired striped block group
    LocatedStripedBlock targetBlockGroup = getBlockGroupAt(target);
    // Update current position
    this.pos = target;
    this.blockEnd = targetBlockGroup.getStartOffset() +
        targetBlockGroup.getBlockSize() - 1;
    currentLocatedBlock = targetBlockGroup;
  }

  @Override
  public synchronized void close() throws IOException {

View on GitHub (pinned to 2add963021)

Solutions

1. Break the read loop on the first read() == -1 and never call read again after it.
2. Before positional reads, validate position < fs.getFileStatus(path).getLen() (re-stat, do not use a cached length).
3. If truncation races are expected, catch this IOException at the top level, re-stat the file, and treat the shorter length as the source of truth.
4. Prefer pread/readFully with range checks computed from a fresh length over raw byte-at-a-time loops on EC files.

Example fix

// before
int n;
while ((n = in.read(buf, 0, buf.length)) != 0) { // wrong sentinel: -1 keeps looping
  sink.write(buf, 0, n);
}

// after
int n;
while ((n = in.read(buf, 0, buf.length)) != -1) {
  sink.write(buf, 0, n);
}

long fileLen = fs.getFileStatus(path).getLen(); // fresh
long remaining = fileLen - position;
if (remaining <= 0) return DONE; // at/after EOF on an EC file - nothing to read
if (length > remaining) length = (int) remaining;
int n = stripedIn.read(position, buf, 0, length);

try {
  n = in.read(buf, 0, buf.length);
} catch (IOException e) {
  if (String.valueOf(e.getMessage()).contains("Attempted to read past end of file")) {
    long freshLen = fs.getFileStatus(path).getLen();
    if (position >= freshLen) return DONE; // genuine EOF (file truncated): stop cleanly
    in.seek(Math.min(position, freshLen - 1)); continue; // shrank mid-read: clamp and retry
  }
  throw e;
}

When it happens

Trigger: Continuing to read() a striped file after a previous read already returned -1 (loop not honoring EOF); positional pread(position, ...) with position >= file length on an EC file; skip() clamped arithmetic feeding an out-of-range target back into blockSeekTo; racing truncation shrinking the EC file between length check and read.

Common situations: Custom record readers / parsers run over erasure-coded datasets (e.g. EC-enabled zones in Hive/Spark warehouses) that ignore the -1 EOF sentinel; readers using a cached file length against files concurrently compacted/truncated; code tested only on replicated files where the equivalent mistake throws a different (caught) exception type.

Related errors

• Cannot seek after EOF
• Could not find target position {}
• Attempted to read past end of file
• End of file reached before reading fully.
• Stream closed

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7f53af5e5e3a65b3. Report an issue: GitHub.