apache/flink · error · ParquetDecodingException

Failed to read from input stream

Error message

Failed to read from input stream

What it means

Thrown by Flink's Parquet vectorized reader when the underlying RLE/bit-packing decoder (used for definition levels, repetition levels, or dictionary-encoded values) hits an IOException while reading bytes from the column page input stream. The ParquetDecodingException wraps the original IOException, so the real cause is always in the 'caused by' chain. It almost always means the byte stream ended prematurely, the page data is corrupt, or the column encoding cannot be decoded with the current reader implementation.

Source

Thrown at flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/vector/reader/RunLengthDecoder.java:286

                        if (buffer.hasArray()) {
                            // byte array has better performance than ByteBuffer
                            this.packer.unpack8Values(
                                    buffer.array(),
                                    buffer.arrayOffset() + buffer.position(),
                                    this.currentBuffer,
                                    valueIndex);
                        } else {
                            this.packer.unpack8Values(
                                    buffer, buffer.position(), this.currentBuffer, valueIndex);
                        }
                        valueIndex += 8;
                    }
                    return;
                default:
                    throw new ParquetDecodingException("not a valid mode " + this.mode);
            }
        } catch (IOException e) {
            throw new ParquetDecodingException("Failed to read from input stream", e);
        }
    }

    enum MODE {
        RLE,
        PACKED
    }
}

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Check the wrapped cause: catch ParquetDecodingException and inspect getCause() — EOFException means truncated file, CRC/codec errors mean corruption.
2. Verify the file is complete and readable with an external tool (parquet-tools / 'parquet cat') on the same storage.
3. Re-produce or re-upload the affected Parquet file; if the producer was interrupted mid-write, regenerate it.
4. If the file is valid, test with a recent parquet-mr writer/reader combination and check Flink JIRAs for encoding-specific reader bugs; report with a reproducer.
5. As a job-level guard, skip bad files/splits via a custom reader wrapper or isolate the failing file by binary-searching the input set.

Example fix

// before: silently failing task
try {
    columnReader.readBatch();
} catch (ParquetDecodingException e) {
    throw e;
}
// after: surface root cause
try {
    columnReader.readBatch();
} catch (ParquetDecodingException e) {
    Throwable root = e.getCause();
    LOG.error("Parquet decode failed, likely corrupt/truncated page: {}", root, e);
    throw e;
}

// Validate file integrity before reading (best-effort)
ParquetFileReader reader = ParquetFileReader.open(conf, path);
if (reader.getFooter().getBlocks().isEmpty()) {
    throw new IOException("Empty/corrupt parquet footer: " + path);
}

try {
    vectorizedReader.readBatch();
} catch (ParquetDecodingException e) {
    if (e.getCause() instanceof EOFException) {
        // truncated file: quarantine and skip, or fail the split
        LOG.warn("Truncated parquet split: {}", split, e);
        throw e;
    }
    throw e;
}

When it happens

Trigger: Reading a Parquet file whose column chunk pages are truncated or corrupt (interrupted write, truncated FTP/HDFS upload); reading a file with a dictionary/RLE encoding variant the reader cannot decode; reading from a split whose byte range is wrong (bad InputSplit offsets); a schema/encoding produced by a Parquet writer version incompatible with the bundled parquet-mr.

Common situations: Jobs reading files that were still being written when the query started; files copied incompletely; compressed blocks damaged by transfer; Parquet files written by Spark/Impala/Hive with encodings the Flink parquet reader mishandles; FS connectors with wrong file size metadata.

Related errors

• Error while waiting for job to be initialized
• Could not build the program from JAR file: {}
• Could not cancel job {}.
• Missing JobID. Specify a JobID to cancel a job.
• Failed to dispose the savepoint '{}'.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/548fcf37f6909b0d. Report an issue: GitHub.