prestodb/presto · error · ParquetDecodingException

Corrupted Parquet file: extra %d values to be consumed when

Error message

Corrupted Parquet file: extra %d values to be consumed when scanning current batch

What it means

Thrown by LongDecimalFlatBatchReader.readWithoutNull when a required fixed_len_byte_array(16) DECIMAL column chunk lacks enough raw values to fill the requested batch. The loop consumed every remaining page but still has remainingInBatch values to read, so the declared value count disagrees with the actual page data. The library classifies this as file corruption and fails rather than emitting a partially-filled Int128ArrayBlock.

Source

Thrown at presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/LongDecimalFlatBatchReader.java:223

        int remainingInBatch = nextBatchSize;
        int startOffset = 0;
        while (remainingInBatch > 0) {
            if (remainingCountInPage == 0) {
                if (!readNextPage()) {
                    break;
                }
            }

            int chunkSize = Math.min(remainingCountInPage, remainingInBatch);

            valuesDecoder.readNext(values, startOffset, chunkSize);
            startOffset += chunkSize;
            remainingInBatch -= chunkSize;
            remainingCountInPage -= chunkSize;
        }

        if (remainingInBatch != 0) {
            throw new ParquetDecodingException(format("Corrupted Parquet file: extra %d values to be consumed when scanning current batch", remainingInBatch));
        }

        Block block = new Int128ArrayBlock(nextBatchSize, Optional.empty(), values);
        return new ColumnChunk(block, new int[0], new int[0]);
    }

    private void seek()
            throws IOException
    {
        if (readOffset == 0) {
            return;
        }

        int remainingInBatch = readOffset;
        int startOffset = 0;
        while (remainingInBatch > 0) {
            if (remainingCountInPage == 0) {
                if (!readNextPage()) {

View on GitHub (pinned to 55bb57d202)

Solutions

1. Re-copy or re-download the file and verify checksums/sizes.
2. Re-run or repair the producing job; confirm it closed the Parquet file writer correctly.
3. Validate row-group/page value counts with parquet-tools or equivalent.
4. Upgrade/fix the external writer if it emits incorrect page headers.
5. Exclude the corrupt row group/split from the scan.

Example fix

// before
ColumnChunk chunk = reader.readNext(timezone); // ParquetDecodingException: extra N values

// after: sanity-check before scanning
long expected = columnChunk.getMeta().getValueCount();
if (chunkByteSize < expected * 16L /* bytes per decimal */) { quarantine(path); return; }
ColumnChunk chunk = reader.readNext(timezone);

long valueCount = colMeta.getValueCount();
long bytesNeeded = valueCount * 16L; // INT128 decimal
if (colChunkLength < bytesNeeded) {
    throw new IOException("Required decimal chunk truncated: " + path);
}

try {
    return reader.readNext(timezone);
} catch (ParquetDecodingException e) {
    quarantineFile(path, e);
    throw new PrestoException(PARQUET_BAD_DATA, "Skipping corrupt decimal column chunk", e);
}

When it happens

Trigger: readNext() on a required DECIMAL(38,x) column when column chunk pages contain fewer values than the batch/metadata requires — readNextPage() exhausts the chunk and remainingInBatch != 0.

Common situations: Files truncated mid-write or by incomplete S3 multipart uploads, corruption during HDFS block transfer, non-conformant third-party Parquet writers with wrong valueCount in page headers, mixing files written by incompatible library versions into one table.

Related errors

• Corrupted Parquet file: extra %d values to be consumed when
• We didn't read correct number of definitionLevels
• Still remaining to be read in current batch.
• Corrupted Parquet file: extra %d values to be consumed when
• Still remaining to be read in current batch.

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/8ae7287b071d9986. Report an issue: GitHub.