prestodb/presto · error · ParquetDecodingException

Corrupted Parquet file: extra %d values to be consumed when

Error message

Corrupted Parquet file: extra %d values to be consumed when scanning current batch

What it means

Int32FlatBatchReader.readWithoutNull throws this when a required int32 column's pages did not deliver all nextBatchSize values during the scanning loop. Because required columns have exactly one value per row, a shortfall means the file's page data contradicts its metadata — treated as corruption.

Source

Thrown at presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/Int32FlatBatchReader.java:222

        int remainingInBatch = nextBatchSize;
        int startOffset = 0;
        while (remainingInBatch > 0) {
            if (remainingCountInPage == 0) {
                if (!readNextPage()) {
                    break;
                }
            }

            int chunkSize = Math.min(remainingCountInPage, remainingInBatch);

            valuesDecoder.readNext(values, startOffset, chunkSize);
            startOffset += chunkSize;
            remainingInBatch -= chunkSize;
            remainingCountInPage -= chunkSize;
        }

        if (remainingInBatch != 0) {
            throw new ParquetDecodingException(format("Corrupted Parquet file: extra %d values to be consumed when scanning current batch", remainingInBatch));
        }

        Block block = new IntArrayBlock(nextBatchSize, Optional.empty(), values);
        return new ColumnChunk(block, new int[0], new int[0]);
    }

    private void seek()
            throws IOException
    {
        if (readOffset == 0) {
            return;
        }

        int remainingInBatch = readOffset;
        int startOffset = 0;
        while (remainingInBatch > 0) {
            if (remainingCountInPage == 0) {
                if (!readNextPage()) {

View on GitHub (pinned to 55bb57d202)

Solutions

Run parquet-tools validation on the file and regenerate it if the chunk is short
Audit the writer's page accounting (valueCount vs actual encoded values)
Restore a good copy of the file; check for partial uploads on S3
Upgrade Presto for improved corruption handling

Example fix

// before
reader.read(batchSize); // throws when file truncated
// after
if (verifyColumnChunkComplete(file, chunk)) {
    reader.read(batchSize);
}

Defensive patterns

Strategy: try-catch

Validate before calling

if (columnDescriptor.isOptional() == false && batchSize > columnChunkMetaData.getValueCount()) {
    throw new IllegalStateException("required column chunk too small for batch");
}

Try / catch

try {
    reader.readNext();
} catch (ParquetDecodingException e) {
    quarantineFileAndAlert();
}

Prevention

Verify object size matches expectation before reading
Test custom writers with parquet-mr reference reader
Enable storage-side checksums (S3 ETag, HDFS CRC)
Regenerate files whose page headers disagree with data

When it happens

Trigger: readNext on a required int32 column whose pages collectively contain fewer values than the requested batch size, detected when remainingInBatch != 0 after the while loop in readWithoutNull.

Common situations: Files truncated mid-chunk; page headers with wrong valueCount from a buggy writer; corrupted/re-chunked files; partial multi-part uploads to object storage.

Related errors

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/cbfc432ee3ed0ea6. Report an issue: GitHub.