What causes "Premature EOF from inputStream" in apache/hadoop?

Run 'hdfs fsck /file -files -blocks -locations' to identify missing or truncated blocks.

apache/hadoop · error · IOException

Premature EOF from inputStream

Error message

Premature EOF from inputStream

What it means

BlockReaderUtil.readFully loops until the requested byte count is filled; a read() returning -1 before then means the block stream ended early — the block has fewer readable bytes than the NameNode's metadata promises. It is the classic truncated-block signature.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderUtil.java:52

    int n = 0;
    for (;;) {
      int nread = reader.read(buf, offset + n, len - n);
      if (nread <= 0)
        return (n == 0) ? nread : n;
      n += nread;
      if (n >= len)
        return n;
    }
  }

  /* See {@link BlockReader#readFully(byte[], int, int)} */
  public static void readFully(BlockReader reader,
      byte[] buf, int off, int len) throws IOException {
    int toRead = len;
    while (toRead > 0) {
      int ret = reader.read(buf, off, toRead);
      if (ret < 0) {
        throw new IOException("Premature EOF from inputStream");
      }
      toRead -= ret;
      off += ret;
    }
  }
}

View on GitHub (pinned to 2add963021)

Solutions

Run 'hdfs fsck /file -files -blocks -locations' to identify missing or truncated blocks.
Retry after re-replication heals the block (or trigger healing by restarting the affected datanode / using fsck -delete).
For concurrently-written files, read only up to the last finalized length or use a commit marker.
If no replica has the bytes, restore the file from source or snapshot — the data does not exist in HDFS.

Defensive patterns

Strategy: retry

Try / catch

long pos = 0;
for (int attempt = 1; attempt <= 3; attempt++) {
  try (FSDataInputStream in = dfs.open(path)) {
    in.seek(pos);
    return readRest(in);
  } catch (IOException e) {
    if (attempt == 3 || !isPrematureEof(e)) throw e;
    // give NN time to re-replicate, then resume from last good position
    sleepBackoff(attempt);
    pos = lastGoodPosition();
  }
}

Prevention

Wrap HDFS reads in position-tracking retry loops so truncated replicas are re-read after re-replication.
For files being written concurrently, bound reads to the last finalized length or a commit marker.
React to premature EOFs with 'hdfs fsck' — they usually mean a replica is physically truncated.

When it happens

Trigger: Reading a block whose replica on the serving datanode is shorter than the recorded block length: data lost after datanode crashes, under-written or corrupted replicas, or reading files whose writer failed mid-commit.

Common situations: Files left inconsistent after writer crashes; failing disks truncating block files; consumers reading files a producer is still writing (length advanced but block not finalized); metadata/block mismatch after a NameNode restore.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/88fa8d643714c0d5. Report an issue: GitHub.