prestodb/presto · error · ParquetDecodingException

Still remaining to be read in current batch.

Error message

Still remaining to be read in current batch.

What it means

Int32FlatBatchReader.readWithNull throws this when, after exhausting the column chunk's pages, the requested batch of nextBatchSize int32 values could not be fully filled. It indicates page value counts (and nullability data) are inconsistent with the requested row count — the metadata promises more values than the pages contain.

Source

Thrown at presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/Int32FlatBatchReader.java:187

                int valueDestinationIndex = startOffset + chunkSize - 1;
                int valueSourceIndex = startOffset + nonNullCount - 1;

                while (valueDestinationIndex >= startOffset) {
                    if (!isNull[valueDestinationIndex]) {
                        values[valueDestinationIndex] = values[valueSourceIndex];
                        valueSourceIndex--;
                    }
                    valueDestinationIndex--;
                }
            }

            startOffset += chunkSize;
            remainingInBatch -= chunkSize;
            remainingCountInPage -= chunkSize;
        }

        if (remainingInBatch != 0) {
            throw new ParquetDecodingException("Still remaining to be read in current batch.");
        }

        if (totalNonNullCount == 0) {
            Block block = RunLengthEncodedBlock.create(field.getType(), null, nextBatchSize);
            return new ColumnChunk(block, new int[0], new int[0]);
        }

        boolean hasNoNull = totalNonNullCount == nextBatchSize;
        Block block = new IntArrayBlock(nextBatchSize, hasNoNull ? Optional.empty() : Optional.of(isNull), values);
        return new ColumnChunk(block, new int[0], new int[0]);
    }

    private ColumnChunk readWithoutNull()
            throws IOException
    {
        int[] values = new int[nextBatchSize];
        int remainingInBatch = nextBatchSize;
        int startOffset = 0;

View on GitHub (pinned to 55bb57d202)

Solutions

1. Validate/regenerate the Parquet file; check with parquet-tools whether the column chunk is complete
2. Fix or upgrade the writer that produced wrong page metadata
3. Upgrade the Presto reader version for better corruption diagnostics
4. Restore a complete copy of the file from object storage checkpoints

Example fix

// before
rows = reader.read(batchSize);
// after
long availableRows = columnChunkRowCount - rowsRead;
rows = reader.read((int) Math.min(batchSize, availableRows));

int readable = (int) Math.min(batchSize, columnChunkMetaData.getValueCount() - rowsRead);
if (readable < batchSize) { requestNewSplit(); }

try {
    reader.readNext();
} catch (ParquetDecodingException e) {
    markSplitFailedAndSkip();
}

When it happens

Trigger: readNext on an optional int32 column where the sum of page valueCounts is less than nextBatchSize; readNextPage() returns null before remainingInBatch reaches 0.

Common situations: Truncated Parquet files; writers emitting wrong page valueCount metadata; corrupted dictionary or RLE bit-packed data causing premature page exhaustion.

Related errors

• We didn't read correct number of definitionLevels
• Still remaining to be read in current batch.
• Corrupted Parquet file: extra %d values to be consumed when
• Corrupted Parquet file: extra %d values to be consumed when
• Still remaining to be read in current batch.

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/e99ead19bb257b89. Report an issue: GitHub.