apache/hadoop · error · EOFException

End of file reached before reading fully.

Error message

End of file reached before reading fully.

What it means

After the initial length check, ByteRangeInputStream.readFully reads in a loop until all requested bytes are transferred. If the underlying HTTP stream returns -1 before nread reaches length, it throws the shared FSExceptionMessages.EOF_IN_READ_FULLY EOFException. This means the server ended the response body earlier than the number of bytes the caller requested.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/ByteRangeInputStream.java:257

  @Override
  public void readFully(long position, byte[] buffer, int offset, int length)
      throws IOException {
    validatePositionedReadArgs(position, buffer, offset, length);
    if (length == 0) {
      return;
    }
    final InputStreamAndFileLength fin = openInputStream(position);
    try {
      if (fin.length != null && length + position > fin.length) {
        throw new EOFException("The length to read " + length
            + " exceeds the file length " + fin.length);
      }
      int nread = 0;
      while (nread < length) {
        int nbytes = fin.in.read(buffer, offset + nread, length - nread);
        if (nbytes < 0) {
          throw new EOFException(FSExceptionMessages.EOF_IN_READ_FULLY);
        }
        nread += nbytes;
      }
    } finally {
      fin.in.close();
    }
  }

  /**
   * Return the current offset from the start of the file
   */
  @Override
  public long getPos() throws IOException {
    return currentPos;
  }

  /**
   * Seeks a different copy of the data.  Returns true if

View on GitHub (pinned to 2add963021)

Solutions

  1. Re-run getFileStatus(path), compare the new length with position + length, and either clamp the read or fail with a clear application error.
  2. Retry the read once on a fresh connection or another NameNode/DataNode if the length still covers the range and the failure looks transient.
  3. Inspect proxy or DataNode logs for truncated responses or reset connections when the reported file length is stable.
  4. Read only committed files, or coordinate truncation/append with readers, when another process may modify the file concurrently.

Example fix

// before
in.readFully(pos, buffer, 0, length);

// after
try {
  in.readFully(pos, buffer, 0, length);
} catch (EOFException e) {
  long currentLen = fs.getFileStatus(path).getLen();
  if (pos + length > currentLen) {
    throw new EOFException("File changed: need " + (pos + length)
        + " bytes, current length is " + currentLen, e);
  }
  throw e;
}
Defensive patterns

Strategy: try-catch

Validate before calling

FileStatus status = fs.getFileStatus(path);
if (position + length > status.getLen()) {
  length = (int) Math.max(0, status.getLen() - position);
}
if (length > 0) {
  in.readFully(position, buffer, offset, length);
}

Try / catch

try {
  in.readFully(position, buffer, offset, length);
} catch (EOFException e) {
  FileStatus latest = fs.getFileStatus(path);
  if (position + length <= latest.getLen()) {
    // Length still covers the range: likely a truncated HTTP response or transient DN failure.
    LOG.warn("Premature EOF for {}; retrying", path, e);
    return readWithRetry(fs, path, position, buffer, offset, length);
  }
  throw e;
}

Prevention

When it happens

Trigger: readFully(position, buffer, offset, length) passes the initial range check, but the response body returns EOF after fewer than length bytes. Common concrete causes are concurrent truncation of the file, a stale server-reported length, an intermediate proxy truncating the range response, or a DataNode stream that fails during transfer.

Common situations: A file is deleted or truncated while a reader is active; a writer exposes an incomplete file before it is committed; an HTTP proxy or load balancer imposes a response-size limit; a DataNode becomes unhealthy during a large positioned read.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/817fc6a2838ef595. Report an issue: GitHub.