{"record":{"id":"56c92635727912a9","repo":"prestodb/presto","slug":"corrupted-parquet-file-extra-d-values-to-be-cons-56c926","errorCode":null,"errorMessage":"Corrupted Parquet file: extra %d values to be consumed when scanning current batch","messagePattern":"Corrupted Parquet file: extra (.+?) values to be consumed when scanning current batch","errorType":"exception","errorClass":"ParquetDecodingException","httpStatus":null,"severity":"error","filePath":"presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/Int64TimeAndTimestampMicrosFlatBatchReader.java","lineNumber":222,"sourceCode":"        int remainingInBatch = nextBatchSize;\n        int startOffset = 0;\n        while (remainingInBatch > 0) {\n            if (remainingCountInPage == 0) {\n                if (!readNextPage()) {\n                    break;\n                }\n            }\n\n            int chunkSize = Math.min(remainingCountInPage, remainingInBatch);\n\n            valuesDecoder.readNext(values, startOffset, chunkSize);\n            startOffset += chunkSize;\n            remainingInBatch -= chunkSize;\n            remainingCountInPage -= chunkSize;\n        }\n\n        if (remainingInBatch != 0) {\n            throw new ParquetDecodingException(format(\"Corrupted Parquet file: extra %d values to be consumed when scanning current batch\", remainingInBatch));\n        }\n\n        Block block = new LongArrayBlock(nextBatchSize, Optional.empty(), values);\n        return new ColumnChunk(block, new int[0], new int[0]);\n    }\n\n    private void seek()\n            throws IOException\n    {\n        if (readOffset == 0) {\n            return;\n        }\n\n        int remainingInBatch = readOffset;\n        int startOffset = 0;\n        while (remainingInBatch > 0) {\n            if (remainingCountInPage == 0) {\n                if (!readNextPage()) {","sourceCodeStart":204,"sourceCodeEnd":240,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/Int64TimeAndTimestampMicrosFlatBatchReader.java#L204-L240","documentation":"Thrown by Int64TimeAndTimestampMicrosFlatBatchReader.readWithoutNull when a required (non-nullable) INT64 time/timestamp-micros column runs out of values before the requested batch (nextBatchSize) is filled. The scan loop consumes all remaining pages of the column chunk, breaks out with values left to read, and the reader concludes the file's declared value counts do not match the actual data. This library treats that mismatch as proof the Parquet file is corrupt or truncated, so it fails fast instead of returning a short or padded block.","triggerScenarios":"Calling readNext on a required INT64 time/timestamp column when the column chunk's pages collectively contain fewer values than page.getValueCount() / the row-group metadata claims — e.g. the loop's readNextPage() returns null before remainingInBatch reaches 0.","commonSituations":"Reading Parquet files truncated by a failed writer or interrupted upload, files produced by buggy/non-conformant Parquet writers with wrong row-group value counts, files corrupted in transfer or stored on flaky storage (HDFS/S3), or version mismatches between writer and reader on page encoding (e.g. v2 data pages, dictionary encoding).","solutions":["Verify the file's integrity (checksum/size) and re-copy or re-download the Parquet file from the source.","Confirm the writing job completed successfully; re-run the producer job to regenerate the file.","Validate the file with a Parquet metadata tool (e.g. parquet-tools meta/rowcounts) to check row-group value counts against actual page data.","If the file is externally produced, check the writer library version for known page-encoding bugs and upgrade or rewrite the data.","Exclude or quarantine the corrupt file/row group from the scan and re-run the query on healthy splits."],"exampleFix":"// before: scanning a truncated file\nColumnChunk chunk = reader.readNext(timezone); // throws ParquetDecodingException\n\n// after: validate metadata first\nParquetMetadata meta = trailer.readFooter(rowGroupMaxPageSize);\nlong totalRows = meta.getBlocks().stream().mapToLong(TableRowGroup::getRowCount).sum();\nif (totalRows < expectedRows) { skipOrReFetchFile(path); return; }","handlingStrategy":"validation","validationCode":"ParquetMetadata meta = readFooter(path);\nlong declaredRows = meta.getBlocks().stream().mapToLong(b -> b.getRowCount()).sum();\nif (declaredRows < expectedRows || fileSizeBytes < minExpectedSize) {\n    throw new IOException(\"File looks truncated: \" + path);\n}","typeGuard":null,"tryCatchPattern":"try {\n    ColumnChunk c = reader.readNext(timezone);\n} catch (ParquetDecodingException e) {\n    log.error(\"Corrupt parquet chunk, skipping split\", e);\n    metrics.corruptFileCounter.increment();\n    return emptyChunk();\n}","preventionTips":["Checksum Parquet files (e.g. md5 in manifest) at write time and verify before scanning.","Ensure writer jobs close ParquetWriter instances (try-with-resources) so footers and counts are complete.","Validate files with parquet-tools after production and before table registration.","Avoid writing directly to final table locations; stage-then-commit instead."],"tags":["parquet","corrupt-file","decoding","batch-reader"],"backgroundTag":"parquet-file-corrupted","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}