prestodb/presto · error · ParquetDecodingException

Still remaining to be read in current batch.

Error message

Still remaining to be read in current batch.

What it means

Thrown by TimestampFlatBatchReader.readWithNull when an optional TIMESTAMP column chunk contains fewer definition-level values than the requested batch. After exhausting all data pages of the chunk (readNextPage() returned null), remainingInBatch is still > 0, which a conformant writer would never produce since definition levels must match the declared value count. The reader reports this as corruption via ParquetDecodingException.

Source

Thrown at presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/TimestampFlatBatchReader.java:187

                int valueDestinationIndex = startOffset + chunkSize - 1;
                int valueSourceIndex = startOffset + nonNullCount - 1;

                while (valueDestinationIndex >= startOffset) {
                    if (!isNull[valueDestinationIndex]) {
                        values[valueDestinationIndex] = values[valueSourceIndex];
                        valueSourceIndex--;
                    }
                    valueDestinationIndex--;
                }
            }

            startOffset += chunkSize;
            remainingInBatch -= chunkSize;
            remainingCountInPage -= chunkSize;
        }

        if (remainingInBatch != 0) {
            throw new ParquetDecodingException("Still remaining to be read in current batch.");
        }

        if (totalNonNullCount == 0) {
            Block block = RunLengthEncodedBlock.create(field.getType(), null, nextBatchSize);
            return new ColumnChunk(block, new int[0], new int[0]);
        }

        boolean hasNoNull = totalNonNullCount == nextBatchSize;
        Block block = new LongArrayBlock(nextBatchSize, hasNoNull ? Optional.empty() : Optional.of(isNull), values);
        return new ColumnChunk(block, new int[0], new int[0]);
    }

    private ColumnChunk readWithoutNull(Optional<DateTimeZone> timezone)
            throws IOException
    {
        long[] values = new long[nextBatchSize];
        int remainingInBatch = nextBatchSize;
        int startOffset = 0;

View on GitHub (pinned to 55bb57d202)

Solutions

1. Verify file integrity (size/checksum) and re-fetch from source.
2. Re-run the producing job; ensure the writer closed the file with a valid footer.
3. Validate row-group value counts against page data with parquet-tools.
4. Upgrade the writer library if it emits incorrect page headers for timestamps.
5. Skip/quarantine the corrupt split and re-run on healthy data.

Example fix

// before
ColumnChunk chunk = reader.readNext(timezone); // batch incomplete

// after
long declared = colMeta.getValueCount();
long decodable = estimatedDefinitionLevelsInChunk(file, rowGroup, column);
if (decodable < declared) { reFetchOrRepair(path); return; }
ColumnChunk chunk = reader.readNext(timezone);

long valueCount = colMeta.getValueCount();
long bytesNeeded = valueCount * 8L; // int64 timestamp + definition levels
if (colChunkLength < bytesNeeded) {
    throw new IOException("Optional timestamp chunk truncated: " + path);
}

try {
    return reader.readNext(timezone);
} catch (ParquetDecodingException e) {
    log.warn("Corrupt optional timestamp chunk; substituting nulls", e);
    return RunLengthEncodedBlock.create(type, null, nextBatchSize);
}

When it happens

Trigger: readNext(timezone) on an optional TIMESTAMP column when the chunk's pages end before nextBatchSize definition levels are decoded — loop breaks on page exhaustion with remainingInBatch != 0.

Common situations: Files truncated by aborted writers or failed uploads, corrupted blocks in HDFS/S3, non-conformant third-party Parquet writers with wrong valueCount in data page headers, writer/reader version incompatibilities (v2 pages, timestamps written as INT96 vs INT64).

Related errors

• Still remaining to be read in current batch.
• Corrupted Parquet file: extra %d values to be consumed when
• We didn't read correct number of definitionLevels
• Still remaining to be read in current batch.
• Corrupted Parquet file: extra %d values to be consumed when

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/b1369be4dbbabc62. Report an issue: GitHub.