{"record":{"id":"cc14cc1c3e032be6","repo":"prestodb/presto","slug":"corrupted-parquet-file-extra-d-values-to-be-cons-cc14cc","errorCode":null,"errorMessage":"Corrupted Parquet file: extra %d values to be consumed when scanning current batch","messagePattern":"Corrupted Parquet file: extra (.+?) values to be consumed when scanning current batch","errorType":"exception","errorClass":"ParquetDecodingException","httpStatus":null,"severity":"error","filePath":"presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/ShortDecimalFlatBatchReader.java","lineNumber":222,"sourceCode":"        int remainingInBatch = nextBatchSize;\n        int startOffset = 0;\n        while (remainingInBatch > 0) {\n            if (remainingCountInPage == 0) {\n                if (!readNextPage()) {\n                    break;\n                }\n            }\n\n            int chunkSize = Math.min(remainingCountInPage, remainingInBatch);\n\n            valuesDecoder.readNext(values, startOffset, chunkSize);\n            startOffset += chunkSize;\n            remainingInBatch -= chunkSize;\n            remainingCountInPage -= chunkSize;\n        }\n\n        if (remainingInBatch != 0) {\n            throw new ParquetDecodingException(format(\"Corrupted Parquet file: extra %d values to be consumed when scanning current batch\", remainingInBatch));\n        }\n\n        Block block = new LongArrayBlock(nextBatchSize, Optional.empty(), values);\n        return new ColumnChunk(block, new int[0], new int[0]);\n    }\n\n    private void seek()\n            throws IOException\n    {\n        if (readOffset == 0) {\n            return;\n        }\n\n        int remainingInBatch = readOffset;\n        int startOffset = 0;\n        while (remainingInBatch > 0) {\n            if (remainingCountInPage == 0) {\n                if (!readNextPage()) {","sourceCodeStart":204,"sourceCodeEnd":240,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/ShortDecimalFlatBatchReader.java#L204-L240","documentation":"Thrown by ShortDecimalFlatBatchReader.readWithoutNull when a required short DECIMAL column chunk cannot supply the full batch of raw INT64 values. The scan consumed every page of the chunk but remainingInBatch values are still owed, so the declared value count is inconsistent with the actual data. The message includes the outstanding value count, and the library treats this as definitive evidence of a corrupted Parquet file.","triggerScenarios":"readNext() on a required short DECIMAL column when readNextPage() exhausts the column chunk while remainingInBatch > 0, i.e. fewer encoded values exist than the batch/metadata expects.","commonSituations":"Truncated uploads or interrupted copy jobs, disk corruption on worker scratch/HDFS volumes, buggy external writers writing wrong page headers, mixing files from incompatible Parquet writer versions in one table location.","solutions":["Verify and re-transfer the file; compare checksums with the producer.","Re-run the writing job and confirm successful completion before consuming.","Inspect row-group metadata vs actual page data using parquet-tools.","Upgrade or fix the writer library if page headers are malformed.","Exclude the corrupt file/row group from the scan plan."],"exampleFix":"// before\nColumnChunk chunk = reader.readNext(timezone); // extra N values owed\n\n// after: bounds check before read\nlong valueCount = colMeta.getValueCount();\nlong bytesNeeded = valueCount * 8L; // int64 decimal\nif (colChunkLength < bytesNeeded) { flagCorrupt(path); return; }\nColumnChunk chunk = reader.readNext(timezone);","handlingStrategy":"validation","validationCode":"long valueCount = colMeta.getValueCount();\nif (colChunkLength < valueCount * 8L) { // int64 short decimal\n    throw new IOException(\"Required short decimal chunk byte-count too small: \" + path);\n}","typeGuard":null,"tryCatchPattern":"try {\n    return reader.readNext(timezone);\n} catch (ParquetDecodingException e) {\n    metrics.corruptChunk(path);\n    throw new PrestoException(PARQUET_BAD_DATA, \"Corrupt short decimal column chunk\", e);\n}","preventionTips":["Enforce checksum validation at data-ingestion boundaries.","Confirm producer jobs terminate cleanly (check for kill -9 / OOM in writer logs).","Validate new files with parquet-tools before adding to table locations.","Do not hand-edit or concatenate Parquet files post-write."],"tags":["parquet","corrupt-file","decimal","decoding"],"backgroundTag":"parquet-file-corrupted","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}