{"record":{"id":"b1369be4dbbabc62","repo":"prestodb/presto","slug":"still-remaining-to-be-read-in-current-batch-b1369b","errorCode":null,"errorMessage":"Still remaining to be read in current batch.","messagePattern":"Still remaining to be read in current batch\\.","errorType":"exception","errorClass":"ParquetDecodingException","httpStatus":null,"severity":"error","filePath":"presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/TimestampFlatBatchReader.java","lineNumber":187,"sourceCode":"                int valueDestinationIndex = startOffset + chunkSize - 1;\n                int valueSourceIndex = startOffset + nonNullCount - 1;\n\n                while (valueDestinationIndex >= startOffset) {\n                    if (!isNull[valueDestinationIndex]) {\n                        values[valueDestinationIndex] = values[valueSourceIndex];\n                        valueSourceIndex--;\n                    }\n                    valueDestinationIndex--;\n                }\n            }\n\n            startOffset += chunkSize;\n            remainingInBatch -= chunkSize;\n            remainingCountInPage -= chunkSize;\n        }\n\n        if (remainingInBatch != 0) {\n            throw new ParquetDecodingException(\"Still remaining to be read in current batch.\");\n        }\n\n        if (totalNonNullCount == 0) {\n            Block block = RunLengthEncodedBlock.create(field.getType(), null, nextBatchSize);\n            return new ColumnChunk(block, new int[0], new int[0]);\n        }\n\n        boolean hasNoNull = totalNonNullCount == nextBatchSize;\n        Block block = new LongArrayBlock(nextBatchSize, hasNoNull ? Optional.empty() : Optional.of(isNull), values);\n        return new ColumnChunk(block, new int[0], new int[0]);\n    }\n\n    private ColumnChunk readWithoutNull(Optional<DateTimeZone> timezone)\n            throws IOException\n    {\n        long[] values = new long[nextBatchSize];\n        int remainingInBatch = nextBatchSize;\n        int startOffset = 0;","sourceCodeStart":169,"sourceCodeEnd":205,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/TimestampFlatBatchReader.java#L169-L205","documentation":"Thrown by TimestampFlatBatchReader.readWithNull when an optional TIMESTAMP column chunk contains fewer definition-level values than the requested batch. After exhausting all data pages of the chunk (readNextPage() returned null), remainingInBatch is still > 0, which a conformant writer would never produce since definition levels must match the declared value count. The reader reports this as corruption via ParquetDecodingException.","triggerScenarios":"readNext(timezone) on an optional TIMESTAMP column when the chunk's pages end before nextBatchSize definition levels are decoded — loop breaks on page exhaustion with remainingInBatch != 0.","commonSituations":"Files truncated by aborted writers or failed uploads, corrupted blocks in HDFS/S3, non-conformant third-party Parquet writers with wrong valueCount in data page headers, writer/reader version incompatibilities (v2 pages, timestamps written as INT96 vs INT64).","solutions":["Verify file integrity (size/checksum) and re-fetch from source.","Re-run the producing job; ensure the writer closed the file with a valid footer.","Validate row-group value counts against page data with parquet-tools.","Upgrade the writer library if it emits incorrect page headers for timestamps.","Skip/quarantine the corrupt split and re-run on healthy data."],"exampleFix":"// before\nColumnChunk chunk = reader.readNext(timezone); // batch incomplete\n\n// after\nlong declared = colMeta.getValueCount();\nlong decodable = estimatedDefinitionLevelsInChunk(file, rowGroup, column);\nif (decodable < declared) { reFetchOrRepair(path); return; }\nColumnChunk chunk = reader.readNext(timezone);","handlingStrategy":"validation","validationCode":"long valueCount = colMeta.getValueCount();\nlong bytesNeeded = valueCount * 8L; // int64 timestamp + definition levels\nif (colChunkLength < bytesNeeded) {\n    throw new IOException(\"Optional timestamp chunk truncated: \" + path);\n}","typeGuard":null,"tryCatchPattern":"try {\n    return reader.readNext(timezone);\n} catch (ParquetDecodingException e) {\n    log.warn(\"Corrupt optional timestamp chunk; substituting nulls\", e);\n    return RunLengthEncodedBlock.create(type, null, nextBatchSize);\n}","preventionTips":["Validate row-group value counts with parquet-tools as part of ingestion CI.","Ensure writers flush and close cleanly; monitor for aborted write jobs.","Checksum files after upload and before table registration.","Match writer timestamp logical-type settings with reader expectations (INT64 millis/micros vs INT96)."],"tags":["parquet","corrupt-file","timestamp","decoding"],"backgroundTag":"parquet-file-corrupted","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}