apache/hadoop · error · IOException

Premature EOF from inputStream

Error message

Premature EOF from inputStream

What it means

Thrown by IOUtils.readFully(InputStream, byte[], int, int) when in.read() returns -1 (end of stream) before the requested 'len' bytes have been read. The method loops until the buffer is full; hitting EOF mid-loop means the stream is shorter than the caller promised, so the read is treated as a hard failure rather than returning a partial count.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java:216

  }

  /**
   * Reads len bytes in a loop.
   *
   * @param in InputStream to read from
   * @param buf The buffer to fill
   * @param off offset from the buffer
   * @param len the length of bytes to read
   * @throws IOException if it could not read requested number of bytes 
   * for any reason (including EOF)
   */
  public static void readFully(InputStream in, byte[] buf,
      int off, int len) throws IOException {
    int toRead = len;
    while (toRead > 0) {
      int ret = in.read(buf, off, toRead);
      if (ret < 0) {
        throw new IOException( "Premature EOF from inputStream");
      }
      toRead -= ret;
      off += ret;
    }
  }
  
  /**
   * Similar to readFully(). Skips bytes in a loop.
   * @param in The InputStream to skip bytes from
   * @param len number of bytes to skip.
   * @throws IOException if it could not skip requested number of bytes 
   * for any reason (including EOF)
   */
  public static void skipFully(InputStream in, long len) throws IOException {
    long amt = len;
    while (amt > 0) {
      long ret = in.skip(amt);
      if (ret == 0) {

View on GitHub (pinned to 2add963021)

Solutions

1. Verify the source length before reading: compare fs.getFileStatus(path).getLen() against the expected byte count.
2. If the file is truncated, re-fetch or regenerate it (re-run the distcp/upload, re-run the job that wrote it).
3. If both a writer and reader touch the file concurrently, ensure the writer closes/flushes (hflush/hsync) before the reader opens it.
4. If a length field drives the read, validate it against remaining bytes and fail with a clearer error before calling readFully.

Example fix

// before: trusts a length read from the stream and fails opaquely at EOF
byte[] rec = new byte[len];
IOUtils.readFully(in, rec, 0, len);

// after: validate remaining bytes first, so truncation is detected with context
long remaining = fs.getFileStatus(path).getLen() - ((FSDataInputStream) in).getPos();
if (remaining < len) {
  throw new EOFException("File " + path + " truncated: need " + len
      + " bytes, only " + remaining + " left");
}
IOUtils.readFully(in, rec, 0, len);

long remaining = fileLen - ((FSDataInputStream) in).getPos();
if (remaining < len) {
  throw new EOFException(path + " truncated: need " + len + ", have " + remaining);
}
IOUtils.readFully(in, buf, off, len);

try {
  IOUtils.readFully(in, buf, off, len);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("Premature EOF")) {
    handleTruncatedFile(path); // refetch / regenerate / skip
  } else { throw e; }
}

When it happens

Trigger: Any readFully() call on a stream that ends early: reading a fixed-length record or header from a truncated file, reading a checksummed block where the checksummed length exceeds the data length, or deserializing a Writable (readFields uses readFully) from bytes shorter than its serialized form.

Common situations: Truncated files from interrupted uploads, failed distcp transfers, or a still-open writer that hasn't flushed; HDFS files with a missing/corrupt final block; version skew where the writer's serialized layout is longer than the reader expects; a FSDataInputStream positioned past the intended segment.

Related errors

• Premature EOF from inputStream after skipping {len-amt} byte
• Attempted to seek or read past the end of the file
• Cannot seek after EOF
• Range starts beyond the file length ({l}): {last}
• Attempted to seek or read past the end of the file

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/0a21c1937741ae33. Report an issue: GitHub.