prestodb/presto · error · ParquetDecodingException

Still remaining to be read in current batch.

Error message

Still remaining to be read in current batch.

What it means

Int64FlatBatchReader.readWithNull throws this when the batch of nextBatchSize int64 values could not be fully read after consuming all pages of the column chunk. Page metadata promised more values (counting nulls via definition levels) than the pages actually contained.

Source

Thrown at presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/Int64FlatBatchReader.java:187

                int valueDestinationIndex = startOffset + chunkSize - 1;
                int valueSourceIndex = startOffset + nonNullCount - 1;

                while (valueDestinationIndex >= startOffset) {
                    if (!isNull[valueDestinationIndex]) {
                        values[valueDestinationIndex] = values[valueSourceIndex];
                        valueSourceIndex--;
                    }
                    valueDestinationIndex--;
                }
            }

            startOffset += chunkSize;
            remainingInBatch -= chunkSize;
            remainingCountInPage -= chunkSize;
        }

        if (remainingInBatch != 0) {
            throw new ParquetDecodingException("Still remaining to be read in current batch.");
        }

        if (totalNonNullCount == 0) {
            Block block = RunLengthEncodedBlock.create(field.getType(), null, nextBatchSize);
            return new ColumnChunk(block, new int[0], new int[0]);
        }

        boolean hasNoNull = totalNonNullCount == nextBatchSize;
        Block block = new LongArrayBlock(nextBatchSize, hasNoNull ? Optional.empty() : Optional.of(isNull), values);
        return new ColumnChunk(block, new int[0], new int[0]);
    }

    private ColumnChunk readWithoutNull()
            throws IOException
    {
        long[] values = new long[nextBatchSize];
        int remainingInBatch = nextBatchSize;
        int startOffset = 0;

View on GitHub (pinned to 55bb57d202)

Solutions

1. Validate and regenerate the file (parquet-tools or full table rewrite)
2. Fix the writer's page value-count accounting if the file is internally produced
3. Upgrade Presto to a version with stricter/earlier corruption detection
4. Re-download the file; look for truncated object-storage writes

Example fix

// before
reader.read(batchSize);
// after
int safe = (int) Math.min(batchSize, remainingRowsInChunk);
reader.read(safe);

long avail = columnChunkMetaData.getValueCount() - rowsRead;
if (batchSize > avail) { batchSize = (int) avail; }

try {
    reader.readNext();
} catch (ParquetDecodingException e) {
    failoverOrRescanFromGoodCopy();
}

When it happens

Trigger: readNext on an optional int64 column where total page valueCounts < nextBatchSize; readNextPage() exhausts all pages while remainingInBatch > 0.

Common situations: Truncated column chunks; writer bugs producing wrong page valueCount; corrupted definition-level (RLE) data ending early.

Related errors

• We didn't read correct number of definitionLevels
• Still remaining to be read in current batch.
• Corrupted Parquet file: extra %d values to be consumed when
• Still remaining to be read in current batch.
• Corrupted Parquet file: extra %d values to be consumed when

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/771c3caf5e9f39ab. Report an issue: GitHub.