prestodb/presto · error · IllegalStateException

We didn't read correct number of definitionLevels

Error message

We didn't read correct number of definitionLevels

What it means

readDefinitionLevels loops reading chunks of definition levels until the whole batch is consumed; if after all pages it has consumed fewer (or more) values than the batch size, remainingInBatch != 0 and it throws IllegalStateException. This means the encoded page data for the column chunk ended before the declared number of values was delivered — the chunk is internally inconsistent (corrupt or written by an incompatible encoder).

Source

Thrown at presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/AbstractNestedBatchReader.java:219

        return repetitionLevelDecodingContext;
    }

    protected final DefinitionLevelDecodingContext readDefinitionLevels(List<DefinitionLevelValuesDecoderContext> decoderInfos, int batchSize)
            throws IOException
    {
        DefinitionLevelDecodingContext definitionLevelDecodingContext = new DefinitionLevelDecodingContext();

        int[] definitionLevels = new int[batchSize];
        int remainingInBatch = batchSize;
        for (DefinitionLevelValuesDecoderContext decoderInfo : decoderInfos) {
            int readChunkSize = decoderInfo.getEnd() - decoderInfo.getStart();
            decoderInfo.getDefinitionLevelDecoder().readNext(definitionLevels, decoderInfo.getStart(), readChunkSize);
            definitionLevelDecodingContext.add(new ValuesDecoderContext(decoderInfo.getValuesDecoder(), decoderInfo.getStart(), decoderInfo.getEnd()));
            remainingInBatch -= readChunkSize;
        }

        if (remainingInBatch != 0) {
            throw new IllegalStateException("We didn't read correct number of definitionLevels");
        }

        definitionLevelDecodingContext.setDefinitionLevels(definitionLevels);
        return definitionLevelDecodingContext;
    }
}

View on GitHub (pinned to 55bb57d202)

Solutions

1. Run parquet-tools/metadata dump on the file to verify the column chunk's page value counts vs actual bytes; restore or rewrite the file if corrupt.
2. Check whether the file was truncated in transfer (compare byte size/checksum with the writer's output) and re-copy it.
3. Confirm the reader's parquet-mr version matches the writer's; a mismatched encoder can produce inconsistent page metadata.
4. If reproducible on a specific file, isolate it and report with the column chunk metadata to the storage/writer owner.

// preflight: verify file size matches last block end position
ParquetMetadata footer = ParquetFileReader.readFooter(conf, new Path(file));
long expectedEnd = footer.getBlocks().get(footer.getBlocks().size() - 1).getEndingPos();
if (fs.getFileStatus(new Path(file)).getLen() < expectedEnd) {
    throw new IllegalStateException("Truncated parquet file: " + file);
}

try {
    reader.readDefinitionLevels(...);
} catch (IllegalStateException e) {
    if (e.getMessage().contains("definitionLevels")) {
        // mark file corrupt, exclude and continue scan / re-run write job
    } else { throw e; }
}

When it happens

Trigger: Reading a nested column chunk where the definition-level decoder returns fewer values than readChunkSize across all pages, leaving remainingInBatch != 0 after the loop.

Common situations: Truncated column chunk data (short files, partial uploads); page metadata (value counts) inconsistent with actual encoded bytes; files written by a buggy or mismatched parquet writer version.

Related errors

• Still remaining to be read in current batch.
• Still remaining to be read in current batch.
• Still remaining to be read in current batch.
• Corrupted Parquet file: extra %d values to be consumed when
• Still remaining to be read in current batch.

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/e52fc247a2cdd5a5. Report an issue: GitHub.