prestodb/presto · error · ParquetDecodingException

Corrupted Parquet file: extra %d values to be consumed when

Error message

Corrupted Parquet file: extra %d values to be consumed when scanning current batch

What it means

Thrown by Int64TimeAndTimestampMicrosFlatBatchReader.readWithoutNull when a required (non-nullable) INT64 time/timestamp-micros column runs out of values before the requested batch (nextBatchSize) is filled. The scan loop consumes all remaining pages of the column chunk, breaks out with values left to read, and the reader concludes the file's declared value counts do not match the actual data. This library treats that mismatch as proof the Parquet file is corrupt or truncated, so it fails fast instead of returning a short or padded block.

Source

Thrown at presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/Int64TimeAndTimestampMicrosFlatBatchReader.java:222

        int remainingInBatch = nextBatchSize;
        int startOffset = 0;
        while (remainingInBatch > 0) {
            if (remainingCountInPage == 0) {
                if (!readNextPage()) {
                    break;
                }
            }

            int chunkSize = Math.min(remainingCountInPage, remainingInBatch);

            valuesDecoder.readNext(values, startOffset, chunkSize);
            startOffset += chunkSize;
            remainingInBatch -= chunkSize;
            remainingCountInPage -= chunkSize;
        }

        if (remainingInBatch != 0) {
            throw new ParquetDecodingException(format("Corrupted Parquet file: extra %d values to be consumed when scanning current batch", remainingInBatch));
        }

        Block block = new LongArrayBlock(nextBatchSize, Optional.empty(), values);
        return new ColumnChunk(block, new int[0], new int[0]);
    }

    private void seek()
            throws IOException
    {
        if (readOffset == 0) {
            return;
        }

        int remainingInBatch = readOffset;
        int startOffset = 0;
        while (remainingInBatch > 0) {
            if (remainingCountInPage == 0) {
                if (!readNextPage()) {

View on GitHub (pinned to 55bb57d202)

Solutions

Verify the file's integrity (checksum/size) and re-copy or re-download the Parquet file from the source.
Confirm the writing job completed successfully; re-run the producer job to regenerate the file.
Validate the file with a Parquet metadata tool (e.g. parquet-tools meta/rowcounts) to check row-group value counts against actual page data.
If the file is externally produced, check the writer library version for known page-encoding bugs and upgrade or rewrite the data.
Exclude or quarantine the corrupt file/row group from the scan and re-run the query on healthy splits.

Example fix

// before: scanning a truncated file
ColumnChunk chunk = reader.readNext(timezone); // throws ParquetDecodingException

// after: validate metadata first
ParquetMetadata meta = trailer.readFooter(rowGroupMaxPageSize);
long totalRows = meta.getBlocks().stream().mapToLong(TableRowGroup::getRowCount).sum();
if (totalRows < expectedRows) { skipOrReFetchFile(path); return; }

Defensive patterns

Strategy: validation

Validate before calling

ParquetMetadata meta = readFooter(path);
long declaredRows = meta.getBlocks().stream().mapToLong(b -> b.getRowCount()).sum();
if (declaredRows < expectedRows || fileSizeBytes < minExpectedSize) {
    throw new IOException("File looks truncated: " + path);
}

Try / catch

try {
    ColumnChunk c = reader.readNext(timezone);
} catch (ParquetDecodingException e) {
    log.error("Corrupt parquet chunk, skipping split", e);
    metrics.corruptFileCounter.increment();
    return emptyChunk();
}

Prevention

Checksum Parquet files (e.g. md5 in manifest) at write time and verify before scanning.
Ensure writer jobs close ParquetWriter instances (try-with-resources) so footers and counts are complete.
Validate files with parquet-tools after production and before table registration.
Avoid writing directly to final table locations; stage-then-commit instead.

When it happens

Trigger: Calling readNext on a required INT64 time/timestamp column when the column chunk's pages collectively contain fewer values than page.getValueCount() / the row-group metadata claims — e.g. the loop's readNextPage() returns null before remainingInBatch reaches 0.

Common situations: Reading Parquet files truncated by a failed writer or interrupted upload, files produced by buggy/non-conformant Parquet writers with wrong row-group value counts, files corrupted in transfer or stored on flaky storage (HDFS/S3), or version mismatches between writer and reader on page encoding (e.g. v2 data pages, dictionary encoding).

Related errors

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/56c92635727912a9. Report an issue: GitHub.