prestodb/presto · error · ParquetDecodingException

Still remaining to be read in current batch.

Error message

Still remaining to be read in current batch.

What it means

Thrown by LongDecimalFlatBatchReader.readWithNull when an optional DECIMAL(38,x) column chunk does not contain enough definition-level entries to fill the requested batch. The scan loop exhausts all pages (readNextPage() returned null) while remainingInBatch is still > 0. Because the number of definition levels must equal the declared value count, this mismatch means the file's metadata or page data is corrupt/truncated.

Source

Thrown at presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/LongDecimalFlatBatchReader.java:188

                int valueSourceIndex = startOffset + nonNullCount - 1;

                while (valueDestinationIndex >= startOffset) {
                    if (!isNull[valueDestinationIndex]) {
                        values[valueDestinationIndex * 2 + 1] = values[valueSourceIndex * 2 + 1];
                        values[valueDestinationIndex * 2] = values[valueSourceIndex * 2];
                        valueSourceIndex--;
                    }
                    valueDestinationIndex--;
                }
            }

            startOffset += chunkSize;
            remainingInBatch -= chunkSize;
            remainingCountInPage -= chunkSize;
        }

        if (remainingInBatch != 0) {
            throw new ParquetDecodingException("Still remaining to be read in current batch.");
        }

        if (totalNonNullCount == 0) {
            Block block = RunLengthEncodedBlock.create(field.getType(), null, nextBatchSize);
            return new ColumnChunk(block, new int[0], new int[0]);
        }

        boolean hasNoNull = totalNonNullCount == nextBatchSize;
        Block block = new Int128ArrayBlock(nextBatchSize, hasNoNull ? Optional.empty() : Optional.of(isNull), values);
        return new ColumnChunk(block, new int[0], new int[0]);
    }

    private ColumnChunk readWithoutNull()
            throws IOException
    {
        long[] values = new long[nextBatchSize * 2];
        int remainingInBatch = nextBatchSize;
        int startOffset = 0;

View on GitHub (pinned to 55bb57d202)

Solutions

1. Verify file checksum/size against the source and re-transfer the file.
2. Regenerate the file with a working Parquet writer; check the producer job completed.
3. Run parquet-tools/pqmeta validation to compare declared vs actual value counts per row group.
4. Check the writer library version for data-page-v2 or definition-level encoding bugs and upgrade.
5. Skip the corrupt split/file in the query and continue with healthy data.

Example fix

// before
ColumnChunk chunk = reader.readNext(timezone); // throws: batch shorter than declared

// after: pre-check row group consistency
long declared = rowGroup.getColumnChunk(desc.getPath()).getMeta().getTotalSize();
long available = fileLength - colChunkStartOffset;
if (available < declared) { markFileCorrupt(path); return nullColumnChunk(nextBatchSize); }

long valueCount = rowGroup.getColumnChunk(desc).getMeta().getValueCount();
long bytesNeeded = valueCount * 16L; // fixed_len_byte_array(16)
if (colChunkEndOffset - colChunkStartOffset < bytesNeeded) {
    throw new IOException("Column chunk smaller than declared values: " + path);
}

try {
    return reader.readNext(timezone);
} catch (ParquetDecodingException e) {
    log.warn("Definition-level mismatch, corrupt chunk", e);
    return RunLengthEncodedBlock.create(type, null, nextBatchSize);
}

When it happens

Trigger: readNext() on an optional fixed_len_byte_array(16) column when the column chunk's data pages end before nextBatchSize definition levels have been decoded — the while loop breaks via a null page and remainingInBatch != 0.

Common situations: Truncated files from aborted writes or partial uploads, buggy external Parquet writers emitting wrong page value counts, files re-written with mismatched row-group metadata, corruption in object storage, or reading with a reader that misinterprets v1/v2 data page headers.

Related errors

• We didn't read correct number of definitionLevels
• Still remaining to be read in current batch.
• Still remaining to be read in current batch.
• Corrupted Parquet file: extra %d values to be consumed when
• Still remaining to be read in current batch.

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/5dc67d6e605ba4f7. Report an issue: GitHub.