apache/hadoop · error · EOFException

The length to read ${length} exceeds the file length ${fin.l

Error message

The length to read ${length} exceeds the file length ${fin.length}

What it means

WebHDFS ByteRangeInputStream.readFully(position, buffer, offset, length) first opens an HTTP range stream and obtains the server-reported file length. Before reading, it rejects a request whose position + length exceeds that length, because the full requested byte range cannot exist. This preserves the FSDataInputStream.readFully contract that either all requested bytes are read or an EOFException is thrown.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/ByteRangeInputStream.java:250

    if (length == 0) {
      return 0;
    }
    try (InputStream in = openInputStream(position).in) {
      return in.read(buffer, offset, length);
    }
  }

  @Override
  public void readFully(long position, byte[] buffer, int offset, int length)
      throws IOException {
    validatePositionedReadArgs(position, buffer, offset, length);
    if (length == 0) {
      return;
    }
    final InputStreamAndFileLength fin = openInputStream(position);
    try {
      if (fin.length != null && length + position > fin.length) {
        throw new EOFException("The length to read " + length
            + " exceeds the file length " + fin.length);
      }
      int nread = 0;
      while (nread < length) {
        int nbytes = fin.in.read(buffer, offset + nread, length - nread);
        if (nbytes < 0) {
          throw new EOFException(FSExceptionMessages.EOF_IN_READ_FULLY);
        }
        nread += nbytes;
      }
    } finally {
      fin.in.close();
    }
  }

  /**
   * Return the current offset from the start of the file
   */

View on GitHub (pinned to 2add963021)

Solutions

Call getFileStatus(path) immediately before the positioned read and clamp length to Math.max(0, fileLen - position).
Correct the position arithmetic, especially the final partial record, offset, and boundary conditions at pos == fileLen.
Refresh the file length and retry once when the file may have changed between the status call and the read.
If the file is expected to be appended concurrently, wait until the writer closes it or use a committed/sentinel file before reading the final record.

Example fix

// before
long len = oldStatus.getLen();
in.readFully(pos, buffer, 0, recordSize); // pos + recordSize > len

// after
long len = fs.getFileStatus(path).getLen();
long available = len - pos;
if (available <= 0) {
  throw new EOFException("No bytes at position " + pos + " of " + path);
}
int toRead = (int) Math.min(recordSize, available);
in.readFully(pos, buffer, 0, toRead);

Defensive patterns

Strategy: validation

Validate before calling

FileStatus status = fs.getFileStatus(path);
long available = status.getLen() - position;
if (available <= 0) {
  throw new EOFException("No bytes to read at position " + position + " of " + path);
}
int safeLength = (int) Math.min(length, available);
in.readFully(position, buffer, offset, safeLength);

Try / catch

try {
  in.readFully(position, buffer, offset, length);
} catch (EOFException e) {
  // Refresh the length and distinguish a normal EOF from an unexpected transport failure.
  throw new EOFException("Requested " + (position + length) + " bytes; current file length is "
      + fs.getFileStatus(path).getLen(), e);
}

Prevention

Always calculate bytes remaining from a fresh FileStatus before a positioned full read.
Treat the final partial record explicitly instead of assuming fileLen is a multiple of the record size.
Use committed files when another process may append or truncate during a read.

When it happens

Trigger: Calling readFully(pos, buf, off, len) on a stream opened from a webhdfs:// or swebhdfs:// path where pos + len is greater than the file length reported by the current open response. Typical causes are an off-by-one position, a fixed-record read that starts near EOF, or a length obtained from an earlier getFileStatus after the file was truncated or not appended as expected.

Common situations: Reading fixed-size records or footer structures from small files; using a stale FileStatus length in a concurrent pipeline; a file truncated by another job between listing and reading; tests that assume a file is longer than the bytes actually uploaded.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/acdb802687fc8494. Report an issue: GitHub.