apache/hadoop · error · EOFException

Premature EOF from inputStream after skipping {len-amt} byte

Error message

Premature EOF from inputStream after skipping {len-amt} byte(s).

What it means

Thrown by IOUtils.skipFully(InputStream, long) when skipping hits end of stream. If in.skip() repeatedly returns 0 the method probes with a single read(); a -1 return proves EOF, and it throws EOFException stating exactly how many of the requested bytes were skipped before the stream ran out. The skipped-count in the message localizes where the stream ended.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java:239

  }
  
  /**
   * Similar to readFully(). Skips bytes in a loop.
   * @param in The InputStream to skip bytes from
   * @param len number of bytes to skip.
   * @throws IOException if it could not skip requested number of bytes 
   * for any reason (including EOF)
   */
  public static void skipFully(InputStream in, long len) throws IOException {
    long amt = len;
    while (amt > 0) {
      long ret = in.skip(amt);
      if (ret == 0) {
        // skip may return 0 even if we're not at EOF.  Luckily, we can 
        // use the read() method to figure out if we're at the end.
        int b = in.read();
        if (b == -1) {
          throw new EOFException( "Premature EOF from inputStream after " +
              "skipping " + (len - amt) + " byte(s).");
        }
        ret = 1;
      }
      amt -= ret;
    }
  }
  
  /**
   * Close the Closeable objects and <b>ignore</b> any {@link Throwable} or
   * null pointers. Must only be used for cleanup in exception handlers.
   *
   * @param logger the log to record problems to at debug level. Can be null.
   * @param closeables the objects to close
   */
  public static void cleanupWithLogger(Logger logger,
      java.io.Closeable... closeables) {
    for (java.io.Closeable c : closeables) {

View on GitHub (pinned to 2add963021)

Solutions

1. Check the requested offset+length against the actual file size before skipping (getFileStatus().getLen()).
2. If the underlying file is truncated, restore or re-write it — the index/split pointing past EOF is a symptom, not the cause.
3. If the stream class returns 0 from skip() when data remains (custom/wrapper streams), read into a scratch buffer instead of relying on skip().
4. Catch EOFException specifically — it carries the skipped count, which tells you the exact offset where data ended.

Example fix

// before: blindly trusts an index offset
IOUtils.skipFully(dataIn, position);

// after: bound-check the offset against the real file length first
long fileLen = fs.getFileStatus(dataFile).getLen();
if (position > fileLen) {
  throw new EOFException("Index offset " + position + " beyond EOF " + fileLen
      + " of " + dataFile + " — file is truncated or index is stale");
}
IOUtils.skipFully(dataIn, position);

long remaining = fileLen - ((FSDataInputStream) in).getPos();
if (remaining < bytesToSkip) {
  throw new EOFException(path + " ends before skip target");
}
IOUtils.skipFully(in, bytesToSkip);

try {
  IOUtils.skipFully(in, len);
} catch (EOFException e) {
  // message states how many bytes were skipped — treat as end of usable data
  LOG.warn("stream ended after skip: {}", e.getMessage());
  return Record.EOF;
}

When it happens

Trigger: Calling skipFully() to position past a region that extends beyond the stream's end: seeking to a record offset from an index (e.g. MapFile index positions), skipping block prefixes of a truncated file, or passing a 'len' larger than the file. Also triggered when skip() returns 0 on a stream that cannot skip (some wrapper streams) and the stream is actually at EOF.

Common situations: A MapFile/SequenceFile index referencing offsets beyond a truncated data file (interrupted write, missing HDFS block); reading a file segment [start, start+len) where the split math overshoots the file; input streams wrapped in buffering layers that return 0 from skip near the buffer end combined with real EOF.

Related errors

• Premature EOF from inputStream
• Cannot seek after EOF
• Attempted to seek or read past the end of the file
• Attempted to seek or read past the end of the file " + targe
• Attempted to seek or read past the end of the file

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/93e4d4748ec40e0d. Report an issue: GitHub.