{"record":{"id":"5dc67d6e605ba4f7","repo":"prestodb/presto","slug":"still-remaining-to-be-read-in-current-batch-5dc67d","errorCode":null,"errorMessage":"Still remaining to be read in current batch.","messagePattern":"Still remaining to be read in current batch\\.","errorType":"exception","errorClass":"ParquetDecodingException","httpStatus":null,"severity":"error","filePath":"presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/LongDecimalFlatBatchReader.java","lineNumber":188,"sourceCode":"                int valueSourceIndex = startOffset + nonNullCount - 1;\n\n                while (valueDestinationIndex >= startOffset) {\n                    if (!isNull[valueDestinationIndex]) {\n                        values[valueDestinationIndex * 2 + 1] = values[valueSourceIndex * 2 + 1];\n                        values[valueDestinationIndex * 2] = values[valueSourceIndex * 2];\n                        valueSourceIndex--;\n                    }\n                    valueDestinationIndex--;\n                }\n            }\n\n            startOffset += chunkSize;\n            remainingInBatch -= chunkSize;\n            remainingCountInPage -= chunkSize;\n        }\n\n        if (remainingInBatch != 0) {\n            throw new ParquetDecodingException(\"Still remaining to be read in current batch.\");\n        }\n\n        if (totalNonNullCount == 0) {\n            Block block = RunLengthEncodedBlock.create(field.getType(), null, nextBatchSize);\n            return new ColumnChunk(block, new int[0], new int[0]);\n        }\n\n        boolean hasNoNull = totalNonNullCount == nextBatchSize;\n        Block block = new Int128ArrayBlock(nextBatchSize, hasNoNull ? Optional.empty() : Optional.of(isNull), values);\n        return new ColumnChunk(block, new int[0], new int[0]);\n    }\n\n    private ColumnChunk readWithoutNull()\n            throws IOException\n    {\n        long[] values = new long[nextBatchSize * 2];\n        int remainingInBatch = nextBatchSize;\n        int startOffset = 0;","sourceCodeStart":170,"sourceCodeEnd":206,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/LongDecimalFlatBatchReader.java#L170-L206","documentation":"Thrown by LongDecimalFlatBatchReader.readWithNull when an optional DECIMAL(38,x) column chunk does not contain enough definition-level entries to fill the requested batch. The scan loop exhausts all pages (readNextPage() returned null) while remainingInBatch is still > 0. Because the number of definition levels must equal the declared value count, this mismatch means the file's metadata or page data is corrupt/truncated.","triggerScenarios":"readNext() on an optional fixed_len_byte_array(16) column when the column chunk's data pages end before nextBatchSize definition levels have been decoded — the while loop breaks via a null page and remainingInBatch != 0.","commonSituations":"Truncated files from aborted writes or partial uploads, buggy external Parquet writers emitting wrong page value counts, files re-written with mismatched row-group metadata, corruption in object storage, or reading with a reader that misinterprets v1/v2 data page headers.","solutions":["Verify file checksum/size against the source and re-transfer the file.","Regenerate the file with a working Parquet writer; check the producer job completed.","Run parquet-tools/pqmeta validation to compare declared vs actual value counts per row group.","Check the writer library version for data-page-v2 or definition-level encoding bugs and upgrade.","Skip the corrupt split/file in the query and continue with healthy data."],"exampleFix":"// before\nColumnChunk chunk = reader.readNext(timezone); // throws: batch shorter than declared\n\n// after: pre-check row group consistency\nlong declared = rowGroup.getColumnChunk(desc.getPath()).getMeta().getTotalSize();\nlong available = fileLength - colChunkStartOffset;\nif (available < declared) { markFileCorrupt(path); return nullColumnChunk(nextBatchSize); }","handlingStrategy":"validation","validationCode":"long valueCount = rowGroup.getColumnChunk(desc).getMeta().getValueCount();\nlong bytesNeeded = valueCount * 16L; // fixed_len_byte_array(16)\nif (colChunkEndOffset - colChunkStartOffset < bytesNeeded) {\n    throw new IOException(\"Column chunk smaller than declared values: \" + path);\n}","typeGuard":null,"tryCatchPattern":"try {\n    return reader.readNext(timezone);\n} catch (ParquetDecodingException e) {\n    log.warn(\"Definition-level mismatch, corrupt chunk\", e);\n    return RunLengthEncodedBlock.create(type, null, nextBatchSize);\n}","preventionTips":["Require writers to use try-with-resources so definition levels and footers are fully flushed.","Run parquet-tools validation as a post-write pipeline step.","Verify file sizes/checksums after upload (S3 ETag or explicit md5).","Pin compatible writer/reader library versions across the pipeline."],"tags":["parquet","corrupt-file","decoding","definition-levels"],"backgroundTag":"parquet-file-corrupted","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}