prestodb/presto · error · ParquetDecodingException

Still remaining to be read in current batch.

Error message

Still remaining to be read in current batch.

What it means

Thrown by BooleanFlatBatchReader.readWithNull when, after consuming all available pages, the batch of nextBatchSize values could not be fully filled (remainingInBatch != 0). It means the column chunk's page value counts / definition-level data ran out before the requested batch size was decoded, so the page metadata and actual data are inconsistent with the requested read.

Source

Thrown at presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/BooleanFlatBatchReader.java:187

                int valueDestinationIndex = startOffset + chunkSize - 1;
                int valueSourceIndex = startOffset + nonNullCount - 1;

                while (valueDestinationIndex >= startOffset) {
                    if (!isNull[valueDestinationIndex]) {
                        values[valueDestinationIndex] = values[valueSourceIndex];
                        valueSourceIndex--;
                    }
                    valueDestinationIndex--;
                }
            }

            startOffset += chunkSize;
            remainingInBatch -= chunkSize;
            remainingCountInPage -= chunkSize;
        }

        if (remainingInBatch != 0) {
            throw new ParquetDecodingException("Still remaining to be read in current batch.");
        }

        if (totalNonNullCount == 0) {
            Block block = RunLengthEncodedBlock.create(field.getType(), null, nextBatchSize);
            return new ColumnChunk(block, new int[0], new int[0]);
        }

        boolean hasNoNull = totalNonNullCount == nextBatchSize;
        Block block = new ByteArrayBlock(nextBatchSize, hasNoNull ? Optional.empty() : Optional.of(isNull), values);
        return new ColumnChunk(block, new int[0], new int[0]);
    }

    private ColumnChunk readWithoutNull()
            throws IOException
    {
        byte[] values = new byte[nextBatchSize];
        int remainingInBatch = nextBatchSize;
        int startOffset = 0;

View on GitHub (pinned to 55bb57d202)

Solutions

1. Verify the Parquet file integrity (e.g. parquet-tools dump / read the whole table) and re-obtain or regenerate the file if truncated
2. Check that the row group metadata (numRows / page value counts) matches the file — if produced by a custom writer, fix the writer's page accounting
3. Ensure the reader's requested batch size never exceeds the column chunk's remaining row count; upgrade Presto to a version with stricter page validation
4. If the file lives on object storage, confirm no partial upload / failed write left a truncated object

Example fix

// before (client reading a truncated file)
parquetReader.read(); // throws ParquetDecodingException
// after
if (!fileExistsFully(objectStats, expectedSize)) {
    reDownloadFile();
}
parquetReader.read();

long rowsInChunk = columnChunkMetaData.getValueCount();
if (batchSize > rowsInChunk) { throw new IllegalStateException("batch exceeds chunk values"); }

try {
    ColumnChunk chunk = reader.readNext();
} catch (ParquetDecodingException e) {
    handleCorruptFile(e); // failover / re-read from backup
}

When it happens

Trigger: readNext requests nextBatchSize rows; readWithNull loops pulling chunks from pages, but readNextPage() returns null (no more pages) while remainingInBatch is still > 0 — i.e. total values across the column chunk's pages is less than the batch size implied by row-group metadata.

Common situations: Truncated or corrupt Parquet files where a column chunk's pages were cut short; files written by buggy/older writers whose page valueCount disagrees with actual encoded values; incorrect page header offsets after a corrupted footer or bad checkpoint of the file on HDFS/S3.

Related errors

• We didn't read correct number of definitionLevels
• Corrupted Parquet file: extra %d values to be consumed when
• Still remaining to be read in current batch.
• Corrupted Parquet file: extra %d values to be consumed when
• Still remaining to be read in current batch.

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/b4d26e5b63bcb7b6. Report an issue: GitHub.