{"record":{"id":"520c74f9585df88e","repo":"prestodb/presto","slug":"still-remaining-to-be-read-in-current-batch-520c74","errorCode":null,"errorMessage":"Still remaining to be read in current batch.","messagePattern":"Still remaining to be read in current batch\\.","errorType":"exception","errorClass":"ParquetDecodingException","httpStatus":null,"severity":"error","filePath":"presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/ShortDecimalFlatBatchReader.java","lineNumber":187,"sourceCode":"                int valueDestinationIndex = startOffset + chunkSize - 1;\n                int valueSourceIndex = startOffset + nonNullCount - 1;\n\n                while (valueDestinationIndex >= startOffset) {\n                    if (!isNull[valueDestinationIndex]) {\n                        values[valueDestinationIndex] = values[valueSourceIndex];\n                        valueSourceIndex--;\n                    }\n                    valueDestinationIndex--;\n                }\n            }\n\n            startOffset += chunkSize;\n            remainingInBatch -= chunkSize;\n            remainingCountInPage -= chunkSize;\n        }\n\n        if (remainingInBatch != 0) {\n            throw new ParquetDecodingException(\"Still remaining to be read in current batch.\");\n        }\n\n        if (totalNonNullCount == 0) {\n            Block block = RunLengthEncodedBlock.create(field.getType(), null, nextBatchSize);\n            return new ColumnChunk(block, new int[0], new int[0]);\n        }\n\n        boolean hasNoNull = totalNonNullCount == nextBatchSize;\n        Block block = new LongArrayBlock(nextBatchSize, hasNoNull ? Optional.empty() : Optional.of(isNull), values);\n        return new ColumnChunk(block, new int[0], new int[0]);\n    }\n\n    private ColumnChunk readWithoutNull()\n            throws IOException\n    {\n        long[] values = new long[nextBatchSize];\n        int remainingInBatch = nextBatchSize;\n        int startOffset = 0;","sourceCodeStart":169,"sourceCodeEnd":205,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/ShortDecimalFlatBatchReader.java#L169-L205","documentation":"Thrown by ShortDecimalFlatBatchReader.readWithNull when an optional short DECIMAL (INT64-backed) column chunk has fewer definition-level values than the requested batch. All pages of the chunk were consumed (readNextPage() returned null) while remainingInBatch is still positive, which cannot happen in a well-formed file since definition levels must cover every declared value. The reader therefore declares the file corrupt.","triggerScenarios":"readNext() on an optional short DECIMAL column when the data pages run out of definition levels before filling nextBatchSize — the while loop exits via page exhaustion with remainingInBatch != 0.","commonSituations":"Partially written or truncated files (aborted Spark/Hive/Impala jobs), object storage corruption, wrong page valueCount from third-party writers, files copied with size-altering transformations, mismatched v1/v2 data page interpretation.","solutions":["Verify file size/checksum and re-fetch the file from its source.","Regenerate the file; confirm the writer job finished and closed the footer properly.","Validate value counts per row group with parquet-tools or a metadata dump.","Check the producing writer library for definition-level encoding bugs; upgrade.","Drop or quarantine the corrupt split and re-run the query."],"exampleFix":"// before\nColumnChunk chunk = reader.readNext(timezone); // throws: batch not fully decodable\n\n// after: pre-validate\nif (rowCountFromFooter != rowsActuallyPresent(file, rowGroup)) { repairOrReFetch(path); return; }\nColumnChunk chunk = reader.readNext(timezone);","handlingStrategy":"validation","validationCode":"long valueCount = colMeta.getValueCount();\nif (footerRowCount < expectedRows || colChunkLength < valueCount) {\n    throw new IOException(\"Optional decimal chunk inconsistent with metadata: \" + path);\n}","typeGuard":null,"tryCatchPattern":"try {\n    return reader.readNext(timezone);\n} catch (ParquetDecodingException e) {\n    log.warn(\"Truncated optional decimal chunk; substituting nulls\", e);\n    return RunLengthEncodedBlock.create(type, null, nextBatchSize);\n}","preventionTips":["Verify file integrity after every write/upload; reject incomplete files at ingestion.","Use rename-based commit so only fully written files are listed by the metastore.","Run a periodic corruption scanner (footer + value-count validation) over hot tables.","Keep writer and reader Parquet library versions aligned."],"tags":["parquet","corrupt-file","decoding","definition-levels"],"backgroundTag":"parquet-file-corrupted","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}