prestodb/presto · error · ParquetDecodingException

Still remaining to be read in current batch.

Error message

Still remaining to be read in current batch.

What it means

Thrown by ShortDecimalFlatBatchReader.readWithNull when an optional short DECIMAL (INT64-backed) column chunk has fewer definition-level values than the requested batch. All pages of the chunk were consumed (readNextPage() returned null) while remainingInBatch is still positive, which cannot happen in a well-formed file since definition levels must cover every declared value. The reader therefore declares the file corrupt.

Source

Thrown at presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/ShortDecimalFlatBatchReader.java:187

                int valueDestinationIndex = startOffset + chunkSize - 1;
                int valueSourceIndex = startOffset + nonNullCount - 1;

                while (valueDestinationIndex >= startOffset) {
                    if (!isNull[valueDestinationIndex]) {
                        values[valueDestinationIndex] = values[valueSourceIndex];
                        valueSourceIndex--;
                    }
                    valueDestinationIndex--;
                }
            }

            startOffset += chunkSize;
            remainingInBatch -= chunkSize;
            remainingCountInPage -= chunkSize;
        }

        if (remainingInBatch != 0) {
            throw new ParquetDecodingException("Still remaining to be read in current batch.");
        }

        if (totalNonNullCount == 0) {
            Block block = RunLengthEncodedBlock.create(field.getType(), null, nextBatchSize);
            return new ColumnChunk(block, new int[0], new int[0]);
        }

        boolean hasNoNull = totalNonNullCount == nextBatchSize;
        Block block = new LongArrayBlock(nextBatchSize, hasNoNull ? Optional.empty() : Optional.of(isNull), values);
        return new ColumnChunk(block, new int[0], new int[0]);
    }

    private ColumnChunk readWithoutNull()
            throws IOException
    {
        long[] values = new long[nextBatchSize];
        int remainingInBatch = nextBatchSize;
        int startOffset = 0;

View on GitHub (pinned to 55bb57d202)

Solutions

1. Verify file size/checksum and re-fetch the file from its source.
2. Regenerate the file; confirm the writer job finished and closed the footer properly.
3. Validate value counts per row group with parquet-tools or a metadata dump.
4. Check the producing writer library for definition-level encoding bugs; upgrade.
5. Drop or quarantine the corrupt split and re-run the query.

Example fix

// before
ColumnChunk chunk = reader.readNext(timezone); // throws: batch not fully decodable

// after: pre-validate
if (rowCountFromFooter != rowsActuallyPresent(file, rowGroup)) { repairOrReFetch(path); return; }
ColumnChunk chunk = reader.readNext(timezone);

long valueCount = colMeta.getValueCount();
if (footerRowCount < expectedRows || colChunkLength < valueCount) {
    throw new IOException("Optional decimal chunk inconsistent with metadata: " + path);
}

try {
    return reader.readNext(timezone);
} catch (ParquetDecodingException e) {
    log.warn("Truncated optional decimal chunk; substituting nulls", e);
    return RunLengthEncodedBlock.create(type, null, nextBatchSize);
}

When it happens

Trigger: readNext() on an optional short DECIMAL column when the data pages run out of definition levels before filling nextBatchSize — the while loop exits via page exhaustion with remainingInBatch != 0.

Common situations: Partially written or truncated files (aborted Spark/Hive/Impala jobs), object storage corruption, wrong page valueCount from third-party writers, files copied with size-altering transformations, mismatched v1/v2 data page interpretation.

Related errors

• We didn't read correct number of definitionLevels
• Still remaining to be read in current batch.
• Still remaining to be read in current batch.
• Corrupted Parquet file: extra %d values to be consumed when
• Still remaining to be read in current batch.

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/520c74f9585df88e. Report an issue: GitHub.