apache/flink · error · IOException

Premature EOF from inputStream

Error message

Premature EOF from inputStream

What it means

IOUtils.readFully loops InputStream.read(buf, off, toRead) until len bytes arrive; a negative return signals EOF before the requested count, and it throws IOException("Premature EOF from inputStream"). The contract is all-or-nothing: partial data is treated as corruption, not as a short read.

Source

Thrown at flink-core/src/main/java/org/apache/flink/util/IOUtils.java:124

    // ------------------------------------------------------------------------

    /**
     * Reads len bytes in a loop.
     *
     * @param in The InputStream to read from
     * @param buf The buffer to fill
     * @param off offset from the buffer
     * @param len the length of bytes to read
     * @throws IOException if it could not read requested number of bytes for any reason (including
     *     EOF)
     */
    public static void readFully(final InputStream in, final byte[] buf, int off, final int len)
            throws IOException {
        int toRead = len;
        while (toRead > 0) {
            final int ret = in.read(buf, off, toRead);
            if (ret < 0) {
                throw new IOException("Premature EOF from inputStream");
            }
            toRead -= ret;
            off += ret;
        }
    }

    /**
     * Similar to {@link #readFully(InputStream, byte[], int, int)}. Returns the total number of
     * bytes read into the buffer.
     *
     * @param in The InputStream to read from
     * @param buf The buffer to fill
     * @return The total number of bytes read into the buffer
     * @throws IOException If the first byte cannot be read for any reason other than end of file,
     *     or if the input stream has been closed, or if some other I/O error occurs.
     */
    public static int tryReadFully(final InputStream in, final byte[] buf) throws IOException {
        int totalRead = 0;

View on GitHub (pinned to 2f3c205e92)

Solutions

  1. Verify the producer actually wrote the full payload — compare file sizes / record counts on both sides
  2. Check the stream source for truncation: re-download the file, inspect disk/HDFS health, and validate checksums
  3. If format versions may differ, read a version/length header first and only readFully for the announced size
  4. Handle the exception where partial data is legal (e.g. tail of a mutable file) by reading incrementally instead of readFully

Example fix

// before
byte[] header = new byte[8];
IOUtils.readFully(in, header, 0, 8); // assumes writer always sends 8 bytes

// after
byte[] header = new byte[8];
try {
    IOUtils.readFully(in, header, 0, 8);
} catch (IOException e) {
    throw new IOException("Corrupt record at offset " + offset + ": " + e.getMessage(), e);
}
Defensive patterns

Strategy: try-catch

Validate before calling

if (in.available() < expectedLen && strict) throw new IllegalStateException("Input likely truncated");

Try / catch

try {
    IOUtils.readFully(in, buf, 0, len);
} catch (IOException e) {
    throw new IOException("Corrupt/truncated input, expected " + len + " bytes: " + e.getMessage(), e);
}

Prevention

When it happens

Trigger: Reading a fixed-size record/header (e.g. serialized block, MAGIC bytes, length-prefixed payload) from a stream that ends early: truncated file, network socket closed mid-transfer, or a producer that wrote fewer bytes than the reader expects (version/format mismatch on record length).

Common situations: Truncated checkpoint/savepoint or cache files on HDFS/local disk; interrupted download of a dependency blob; a serializer change so the reader's expected length no longer matches what was written; sockets closed by timeouts during blob fetch.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/3f66d183c461e61a. Report an issue: GitHub.