{"record":{"id":"7bac91f86761cc95","repo":"prestodb/presto","slug":"corrupted-parquet-file-extra-d-values-to-be-cons-7bac91","errorCode":null,"errorMessage":"Corrupted Parquet file: extra %d values to be consumed when scanning current batch","messagePattern":"Corrupted Parquet file: extra (.+?) values to be consumed when scanning current batch","errorType":"exception","errorClass":"ParquetDecodingException","httpStatus":null,"severity":"error","filePath":"presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/TimestampFlatBatchReader.java","lineNumber":222,"sourceCode":"        int remainingInBatch = nextBatchSize;\n        int startOffset = 0;\n        while (remainingInBatch > 0) {\n            if (remainingCountInPage == 0) {\n                if (!readNextPage()) {\n                    break;\n                }\n            }\n\n            int chunkSize = Math.min(remainingCountInPage, remainingInBatch);\n\n            valuesDecoder.readNext(values, startOffset, chunkSize, timezone);\n            startOffset += chunkSize;\n            remainingInBatch -= chunkSize;\n            remainingCountInPage -= chunkSize;\n        }\n\n        if (remainingInBatch != 0) {\n            throw new ParquetDecodingException(format(\"Corrupted Parquet file: extra %d values to be consumed when scanning current batch\", remainingInBatch));\n        }\n\n        Block block = new LongArrayBlock(nextBatchSize, Optional.empty(), values);\n        return new ColumnChunk(block, new int[0], new int[0]);\n    }\n\n    private void seek()\n            throws IOException\n    {\n        if (readOffset == 0) {\n            return;\n        }\n\n        int remainingInBatch = readOffset;\n        int startOffset = 0;\n        while (remainingInBatch > 0) {\n            if (remainingCountInPage == 0) {\n                if (!readNextPage()) {","sourceCodeStart":204,"sourceCodeEnd":240,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/TimestampFlatBatchReader.java#L204-L240","documentation":"Thrown by TimestampFlatBatchReader.readWithoutNull when a required TIMESTAMP column chunk lacks enough raw INT64 values to fill the requested batch. The loop consumed every page of the chunk while values were still owed (remainingInBatch > 0), meaning the declared value count does not match the encoded data. The library labels this 'Corrupted Parquet file' with the outstanding value count and aborts the scan.","triggerScenarios":"readNext(timezone) on a required TIMESTAMP column when readNextPage() exhausts the column chunk while remainingInBatch > 0 — fewer encoded values exist than the batch/metadata demands.","commonSituations":"Truncated or partially uploaded files, interrupted Hive/Spark writes, object-store corruption or bit rot, incompatible writer versions emitting wrong page valueCounts, files manually concatenated or edited after writing.","solutions":["Re-transfer the file and verify checksum/size against the producer.","Regenerate the file with a completed writer job; check job logs for errors.","Validate value counts per page/row group using parquet-tools or a metadata reader.","Fix/upgrade the external writer if page headers are incorrect.","Exclude the corrupt row group/file from the scan and continue with the rest."],"exampleFix":"// before\nColumnChunk chunk = reader.readNext(timezone); // extra N values owed\n\n// after\nlong valueCount = colMeta.getValueCount();\nif (colChunkLength < valueCount * 8L) { quarantine(path); return; }\nColumnChunk chunk = reader.readNext(timezone);","handlingStrategy":"validation","validationCode":"long valueCount = colMeta.getValueCount();\nif (colChunkLength < valueCount * 8L) { // required int64 timestamp\n    throw new IOException(\"Required timestamp chunk byte-count mismatch: \" + path);\n}","typeGuard":null,"tryCatchPattern":"try {\n    return reader.readNext(timezone);\n} catch (ParquetDecodingException e) {\n    quarantineFile(path, e);\n    throw new PrestoException(PARQUET_BAD_DATA, \"Corrupt timestamp column chunk\", e);\n}","preventionTips":["Stage files and atomically publish so truncated writes are never queryable.","Verify checksums/sizes on upload completion (multipart-complete verification).","Run post-write metadata validation comparing declared vs actual value counts.","Standardize on one writer library/version to avoid header inconsistencies."],"tags":["parquet","corrupt-file","timestamp","decoding"],"backgroundTag":"parquet-file-corrupted","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}