{"record":{"id":"e52fc247a2cdd5a5","repo":"prestodb/presto","slug":"we-didn-t-read-correct-number-of-definitionlevels","errorCode":null,"errorMessage":"We didn't read correct number of definitionLevels","messagePattern":"We didn't read correct number of definitionLevels","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/AbstractNestedBatchReader.java","lineNumber":219,"sourceCode":"        return repetitionLevelDecodingContext;\n    }\n\n    protected final DefinitionLevelDecodingContext readDefinitionLevels(List<DefinitionLevelValuesDecoderContext> decoderInfos, int batchSize)\n            throws IOException\n    {\n        DefinitionLevelDecodingContext definitionLevelDecodingContext = new DefinitionLevelDecodingContext();\n\n        int[] definitionLevels = new int[batchSize];\n        int remainingInBatch = batchSize;\n        for (DefinitionLevelValuesDecoderContext decoderInfo : decoderInfos) {\n            int readChunkSize = decoderInfo.getEnd() - decoderInfo.getStart();\n            decoderInfo.getDefinitionLevelDecoder().readNext(definitionLevels, decoderInfo.getStart(), readChunkSize);\n            definitionLevelDecodingContext.add(new ValuesDecoderContext(decoderInfo.getValuesDecoder(), decoderInfo.getStart(), decoderInfo.getEnd()));\n            remainingInBatch -= readChunkSize;\n        }\n\n        if (remainingInBatch != 0) {\n            throw new IllegalStateException(\"We didn't read correct number of definitionLevels\");\n        }\n\n        definitionLevelDecodingContext.setDefinitionLevels(definitionLevels);\n        return definitionLevelDecodingContext;\n    }\n}\n","sourceCodeStart":201,"sourceCodeEnd":226,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/AbstractNestedBatchReader.java#L201-L226","documentation":"readDefinitionLevels loops reading chunks of definition levels until the whole batch is consumed; if after all pages it has consumed fewer (or more) values than the batch size, remainingInBatch != 0 and it throws IllegalStateException. This means the encoded page data for the column chunk ended before the declared number of values was delivered — the chunk is internally inconsistent (corrupt or written by an incompatible encoder).","triggerScenarios":"Reading a nested column chunk where the definition-level decoder returns fewer values than readChunkSize across all pages, leaving remainingInBatch != 0 after the loop.","commonSituations":"Truncated column chunk data (short files, partial uploads); page metadata (value counts) inconsistent with actual encoded bytes; files written by a buggy or mismatched parquet writer version.","solutions":["Run parquet-tools/metadata dump on the file to verify the column chunk's page value counts vs actual bytes; restore or rewrite the file if corrupt.","Check whether the file was truncated in transfer (compare byte size/checksum with the writer's output) and re-copy it.","Confirm the reader's parquet-mr version matches the writer's; a mismatched encoder can produce inconsistent page metadata.","If reproducible on a specific file, isolate it and report with the column chunk metadata to the storage/writer owner."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// preflight: verify file size matches last block end position\nParquetMetadata footer = ParquetFileReader.readFooter(conf, new Path(file));\nlong expectedEnd = footer.getBlocks().get(footer.getBlocks().size() - 1).getEndingPos();\nif (fs.getFileStatus(new Path(file)).getLen() < expectedEnd) {\n    throw new IllegalStateException(\"Truncated parquet file: \" + file);\n}","typeGuard":null,"tryCatchPattern":"try {\n    reader.readDefinitionLevels(...);\n} catch (IllegalStateException e) {\n    if (e.getMessage().contains(\"definitionLevels\")) {\n        // mark file corrupt, exclude and continue scan / re-run write job\n    } else { throw e; }\n}","preventionTips":["Checksum files after transfer to detect truncation before query time.","Run periodic parquet metadata validation jobs over stored data.","Ensure writer jobs complete atomically (no partial uploads visible to readers)."],"tags":["parquet","definition-levels","corrupt-file","decoding"],"backgroundTag":"parquet-page-value-count-mismatch","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}