{"record":{"id":"8ae7287b071d9986","repo":"prestodb/presto","slug":"corrupted-parquet-file-extra-d-values-to-be-cons-8ae728","errorCode":null,"errorMessage":"Corrupted Parquet file: extra %d values to be consumed when scanning current batch","messagePattern":"Corrupted Parquet file: extra (.+?) values to be consumed when scanning current batch","errorType":"exception","errorClass":"ParquetDecodingException","httpStatus":null,"severity":"error","filePath":"presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/LongDecimalFlatBatchReader.java","lineNumber":223,"sourceCode":"        int remainingInBatch = nextBatchSize;\n        int startOffset = 0;\n        while (remainingInBatch > 0) {\n            if (remainingCountInPage == 0) {\n                if (!readNextPage()) {\n                    break;\n                }\n            }\n\n            int chunkSize = Math.min(remainingCountInPage, remainingInBatch);\n\n            valuesDecoder.readNext(values, startOffset, chunkSize);\n            startOffset += chunkSize;\n            remainingInBatch -= chunkSize;\n            remainingCountInPage -= chunkSize;\n        }\n\n        if (remainingInBatch != 0) {\n            throw new ParquetDecodingException(format(\"Corrupted Parquet file: extra %d values to be consumed when scanning current batch\", remainingInBatch));\n        }\n\n        Block block = new Int128ArrayBlock(nextBatchSize, Optional.empty(), values);\n        return new ColumnChunk(block, new int[0], new int[0]);\n    }\n\n    private void seek()\n            throws IOException\n    {\n        if (readOffset == 0) {\n            return;\n        }\n\n        int remainingInBatch = readOffset;\n        int startOffset = 0;\n        while (remainingInBatch > 0) {\n            if (remainingCountInPage == 0) {\n                if (!readNextPage()) {","sourceCodeStart":205,"sourceCodeEnd":241,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/LongDecimalFlatBatchReader.java#L205-L241","documentation":"Thrown by LongDecimalFlatBatchReader.readWithoutNull when a required fixed_len_byte_array(16) DECIMAL column chunk lacks enough raw values to fill the requested batch. The loop consumed every remaining page but still has remainingInBatch values to read, so the declared value count disagrees with the actual page data. The library classifies this as file corruption and fails rather than emitting a partially-filled Int128ArrayBlock.","triggerScenarios":"readNext() on a required DECIMAL(38,x) column when column chunk pages contain fewer values than the batch/metadata requires — readNextPage() exhausts the chunk and remainingInBatch != 0.","commonSituations":"Files truncated mid-write or by incomplete S3 multipart uploads, corruption during HDFS block transfer, non-conformant third-party Parquet writers with wrong valueCount in page headers, mixing files written by incompatible library versions into one table.","solutions":["Re-copy or re-download the file and verify checksums/sizes.","Re-run or repair the producing job; confirm it closed the Parquet file writer correctly.","Validate row-group/page value counts with parquet-tools or equivalent.","Upgrade/fix the external writer if it emits incorrect page headers.","Exclude the corrupt row group/split from the scan."],"exampleFix":"// before\nColumnChunk chunk = reader.readNext(timezone); // ParquetDecodingException: extra N values\n\n// after: sanity-check before scanning\nlong expected = columnChunk.getMeta().getValueCount();\nif (chunkByteSize < expected * 16L /* bytes per decimal */) { quarantine(path); return; }\nColumnChunk chunk = reader.readNext(timezone);","handlingStrategy":"validation","validationCode":"long valueCount = colMeta.getValueCount();\nlong bytesNeeded = valueCount * 16L; // INT128 decimal\nif (colChunkLength < bytesNeeded) {\n    throw new IOException(\"Required decimal chunk truncated: \" + path);\n}","typeGuard":null,"tryCatchPattern":"try {\n    return reader.readNext(timezone);\n} catch (ParquetDecodingException e) {\n    quarantineFile(path, e);\n    throw new PrestoException(PARQUET_BAD_DATA, \"Skipping corrupt decimal column chunk\", e);\n}","preventionTips":["Use staged writes plus atomic commit so partial files never become visible.","Checksum and size-verify every Parquet file before query time.","Validate with parquet-tools in CI after each write job.","Watch for multi-part upload failures on object storage; abort/complete explicitly."],"tags":["parquet","corrupt-file","decimal","decoding"],"backgroundTag":"parquet-file-corrupted","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}