prestodb/presto · error · ParquetDecodingException

Corrupted Parquet file: extra %d values to be consumed when

Error message

Corrupted Parquet file: extra %d values to be consumed when scanning current batch

What it means

Thrown by ShortDecimalFlatBatchReader.readWithoutNull when a required short DECIMAL column chunk cannot supply the full batch of raw INT64 values. The scan consumed every page of the chunk but remainingInBatch values are still owed, so the declared value count is inconsistent with the actual data. The message includes the outstanding value count, and the library treats this as definitive evidence of a corrupted Parquet file.

Source

Thrown at presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/ShortDecimalFlatBatchReader.java:222

        int remainingInBatch = nextBatchSize;
        int startOffset = 0;
        while (remainingInBatch > 0) {
            if (remainingCountInPage == 0) {
                if (!readNextPage()) {
                    break;
                }
            }

            int chunkSize = Math.min(remainingCountInPage, remainingInBatch);

            valuesDecoder.readNext(values, startOffset, chunkSize);
            startOffset += chunkSize;
            remainingInBatch -= chunkSize;
            remainingCountInPage -= chunkSize;
        }

        if (remainingInBatch != 0) {
            throw new ParquetDecodingException(format("Corrupted Parquet file: extra %d values to be consumed when scanning current batch", remainingInBatch));
        }

        Block block = new LongArrayBlock(nextBatchSize, Optional.empty(), values);
        return new ColumnChunk(block, new int[0], new int[0]);
    }

    private void seek()
            throws IOException
    {
        if (readOffset == 0) {
            return;
        }

        int remainingInBatch = readOffset;
        int startOffset = 0;
        while (remainingInBatch > 0) {
            if (remainingCountInPage == 0) {
                if (!readNextPage()) {

View on GitHub (pinned to 55bb57d202)

Solutions

Verify and re-transfer the file; compare checksums with the producer.
Re-run the writing job and confirm successful completion before consuming.
Inspect row-group metadata vs actual page data using parquet-tools.
Upgrade or fix the writer library if page headers are malformed.
Exclude the corrupt file/row group from the scan plan.

Example fix

// before
ColumnChunk chunk = reader.readNext(timezone); // extra N values owed

// after: bounds check before read
long valueCount = colMeta.getValueCount();
long bytesNeeded = valueCount * 8L; // int64 decimal
if (colChunkLength < bytesNeeded) { flagCorrupt(path); return; }
ColumnChunk chunk = reader.readNext(timezone);

Defensive patterns

Strategy: validation

Validate before calling

long valueCount = colMeta.getValueCount();
if (colChunkLength < valueCount * 8L) { // int64 short decimal
    throw new IOException("Required short decimal chunk byte-count too small: " + path);
}

Try / catch

try {
    return reader.readNext(timezone);
} catch (ParquetDecodingException e) {
    metrics.corruptChunk(path);
    throw new PrestoException(PARQUET_BAD_DATA, "Corrupt short decimal column chunk", e);
}

Prevention

Enforce checksum validation at data-ingestion boundaries.
Confirm producer jobs terminate cleanly (check for kill -9 / OOM in writer logs).
Validate new files with parquet-tools before adding to table locations.
Do not hand-edit or concatenate Parquet files post-write.

When it happens

Trigger: readNext() on a required short DECIMAL column when readNextPage() exhausts the column chunk while remainingInBatch > 0, i.e. fewer encoded values exist than the batch/metadata expects.

Common situations: Truncated uploads or interrupted copy jobs, disk corruption on worker scratch/HDFS volumes, buggy external writers writing wrong page headers, mixing files from incompatible Parquet writer versions in one table location.

Related errors

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/cc14cc1c3e032be6. Report an issue: GitHub.