apache/hadoop · error · IOException

Offset: {} exceeds file length: {}

Error message

Offset: {} exceeds file length: {}

What it means

getBlockRange() guards the start of every read range: offset must be strictly below the stream's current file length (completed blocks plus last block being written). Reading at or beyond that throws this IOException. On a healthy file it indicates caller math errors; on a shrinking file it indicates a truncate raced the reader.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java:553

      return locatedBlocks.get(targetBlockIdx);
    }
  }

  /**
   * Get blocks in the specified range.
   * Fetch them from the namenode if not cached. This function
   * will not get a read request beyond the EOF.
   * @param offset starting offset in file
   * @param length length of data
   * @return consequent segment of located blocks
   * @throws IOException
   */
  private List<LocatedBlock> getBlockRange(long offset,
      long length)  throws IOException {
    // getFileLength(): returns total file length
    // locatedBlocks.getFileLength(): returns length of completed blocks
    if (offset >= getFileLength()) {
      throw new IOException("Offset: " + offset +
        " exceeds file length: " + getFileLength());
    }

    synchronized(infoLock) {
      final List<LocatedBlock> blocks;
      final long lengthOfCompleteBlk = locatedBlocks.getFileLength();
      final boolean readOffsetWithinCompleteBlk = offset < lengthOfCompleteBlk;
      final boolean readLengthPastCompleteBlk = offset + length > lengthOfCompleteBlk;

      if (readOffsetWithinCompleteBlk) {
        //get the blocks of finalized (completed) block range
        blocks = getFinalizedBlockRange(offset,
          Math.min(length, lengthOfCompleteBlk - offset));
      } else {
        blocks = new ArrayList<>(1);
      }

      // get the blocks from incomplete block range

View on GitHub (pinned to 2add963021)

Solutions

1. Validate offset+length against a fresh getFileStatus().getLen() before issuing reads
2. Return EOF cleanly for pos >= len instead of calling read at all
3. Re-open the stream after external truncate events to refresh locatedBlocks

long len = fs.getFileStatus(path).getLen();
if (offset >= len) {
  return -1; // EOF by contract
}
// only now issue the read that calls getBlockRange internally

try {
  in.readFully(buf, off, len);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("exceeds file length")) {
    // stale length: refresh and clamp
    long cur = fs.getFileStatus(path).getLen();
    in.readFully(buf, off, (int) Math.min(len, cur - in.getPos()));
  } else throw e;
}

When it happens

Trigger: read(buffer, off, len) or readFully positioned at offset >= getFileLength(); the sequential reader's pos advanced past the file's actual length after concurrent truncation.

Common situations: Split readers in compute frameworks computing offsets from stale file lengths; tailers racing log truncation; unit tests reading empty files with nonzero offsets.

Related errors

• offset < 0 || offset >= getFileLength(), offset={}, locatedB
• Offset {startOffset} and length {length} don't match block
• Could not find target position {}
• Path part {s} from URI {p} is not a valid filename.
• {key} = {v} <= 0

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/10106063a2623818. Report an issue: GitHub.