{"record":{"id":"cbfc432ee3ed0ea6","repo":"prestodb/presto","slug":"corrupted-parquet-file-extra-d-values-to-be-cons-cbfc43","errorCode":null,"errorMessage":"Corrupted Parquet file: extra %d values to be consumed when scanning current batch","messagePattern":"Corrupted Parquet file: extra (.+?) values to be consumed when scanning current batch","errorType":"exception","errorClass":"ParquetDecodingException","httpStatus":null,"severity":"error","filePath":"presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/Int32FlatBatchReader.java","lineNumber":222,"sourceCode":"        int remainingInBatch = nextBatchSize;\n        int startOffset = 0;\n        while (remainingInBatch > 0) {\n            if (remainingCountInPage == 0) {\n                if (!readNextPage()) {\n                    break;\n                }\n            }\n\n            int chunkSize = Math.min(remainingCountInPage, remainingInBatch);\n\n            valuesDecoder.readNext(values, startOffset, chunkSize);\n            startOffset += chunkSize;\n            remainingInBatch -= chunkSize;\n            remainingCountInPage -= chunkSize;\n        }\n\n        if (remainingInBatch != 0) {\n            throw new ParquetDecodingException(format(\"Corrupted Parquet file: extra %d values to be consumed when scanning current batch\", remainingInBatch));\n        }\n\n        Block block = new IntArrayBlock(nextBatchSize, Optional.empty(), values);\n        return new ColumnChunk(block, new int[0], new int[0]);\n    }\n\n    private void seek()\n            throws IOException\n    {\n        if (readOffset == 0) {\n            return;\n        }\n\n        int remainingInBatch = readOffset;\n        int startOffset = 0;\n        while (remainingInBatch > 0) {\n            if (remainingCountInPage == 0) {\n                if (!readNextPage()) {","sourceCodeStart":204,"sourceCodeEnd":240,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/Int32FlatBatchReader.java#L204-L240","documentation":"Int32FlatBatchReader.readWithoutNull throws this when a required int32 column's pages did not deliver all nextBatchSize values during the scanning loop. Because required columns have exactly one value per row, a shortfall means the file's page data contradicts its metadata — treated as corruption.","triggerScenarios":"readNext on a required int32 column whose pages collectively contain fewer values than the requested batch size, detected when remainingInBatch != 0 after the while loop in readWithoutNull.","commonSituations":"Files truncated mid-chunk; page headers with wrong valueCount from a buggy writer; corrupted/re-chunked files; partial multi-part uploads to object storage.","solutions":["Run parquet-tools validation on the file and regenerate it if the chunk is short","Audit the writer's page accounting (valueCount vs actual encoded values)","Restore a good copy of the file; check for partial uploads on S3","Upgrade Presto for improved corruption handling"],"exampleFix":"// before\nreader.read(batchSize); // throws when file truncated\n// after\nif (verifyColumnChunkComplete(file, chunk)) {\n    reader.read(batchSize);\n}","handlingStrategy":"try-catch","validationCode":"if (columnDescriptor.isOptional() == false && batchSize > columnChunkMetaData.getValueCount()) {\n    throw new IllegalStateException(\"required column chunk too small for batch\");\n}","typeGuard":null,"tryCatchPattern":"try {\n    reader.readNext();\n} catch (ParquetDecodingException e) {\n    quarantineFileAndAlert();\n}","preventionTips":["Verify object size matches expectation before reading","Test custom writers with parquet-mr reference reader","Enable storage-side checksums (S3 ETag, HDFS CRC)","Regenerate files whose page headers disagree with data"],"tags":["parquet","decoding","corrupt-file"],"backgroundTag":"parquet-decode-batch-short-read","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}