{"record":{"id":"b4d26e5b63bcb7b6","repo":"prestodb/presto","slug":"still-remaining-to-be-read-in-current-batch","errorCode":null,"errorMessage":"Still remaining to be read in current batch.","messagePattern":"Still remaining to be read in current batch\\.","errorType":"exception","errorClass":"ParquetDecodingException","httpStatus":null,"severity":"error","filePath":"presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/BooleanFlatBatchReader.java","lineNumber":187,"sourceCode":"                int valueDestinationIndex = startOffset + chunkSize - 1;\n                int valueSourceIndex = startOffset + nonNullCount - 1;\n\n                while (valueDestinationIndex >= startOffset) {\n                    if (!isNull[valueDestinationIndex]) {\n                        values[valueDestinationIndex] = values[valueSourceIndex];\n                        valueSourceIndex--;\n                    }\n                    valueDestinationIndex--;\n                }\n            }\n\n            startOffset += chunkSize;\n            remainingInBatch -= chunkSize;\n            remainingCountInPage -= chunkSize;\n        }\n\n        if (remainingInBatch != 0) {\n            throw new ParquetDecodingException(\"Still remaining to be read in current batch.\");\n        }\n\n        if (totalNonNullCount == 0) {\n            Block block = RunLengthEncodedBlock.create(field.getType(), null, nextBatchSize);\n            return new ColumnChunk(block, new int[0], new int[0]);\n        }\n\n        boolean hasNoNull = totalNonNullCount == nextBatchSize;\n        Block block = new ByteArrayBlock(nextBatchSize, hasNoNull ? Optional.empty() : Optional.of(isNull), values);\n        return new ColumnChunk(block, new int[0], new int[0]);\n    }\n\n    private ColumnChunk readWithoutNull()\n            throws IOException\n    {\n        byte[] values = new byte[nextBatchSize];\n        int remainingInBatch = nextBatchSize;\n        int startOffset = 0;","sourceCodeStart":169,"sourceCodeEnd":205,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/BooleanFlatBatchReader.java#L169-L205","documentation":"Thrown by BooleanFlatBatchReader.readWithNull when, after consuming all available pages, the batch of nextBatchSize values could not be fully filled (remainingInBatch != 0). It means the column chunk's page value counts / definition-level data ran out before the requested batch size was decoded, so the page metadata and actual data are inconsistent with the requested read.","triggerScenarios":"readNext requests nextBatchSize rows; readWithNull loops pulling chunks from pages, but readNextPage() returns null (no more pages) while remainingInBatch is still > 0 — i.e. total values across the column chunk's pages is less than the batch size implied by row-group metadata.","commonSituations":"Truncated or corrupt Parquet files where a column chunk's pages were cut short; files written by buggy/older writers whose page valueCount disagrees with actual encoded values; incorrect page header offsets after a corrupted footer or bad checkpoint of the file on HDFS/S3.","solutions":["Verify the Parquet file integrity (e.g. parquet-tools dump / read the whole table) and re-obtain or regenerate the file if truncated","Check that the row group metadata (numRows / page value counts) matches the file — if produced by a custom writer, fix the writer's page accounting","Ensure the reader's requested batch size never exceeds the column chunk's remaining row count; upgrade Presto to a version with stricter page validation","If the file lives on object storage, confirm no partial upload / failed write left a truncated object"],"exampleFix":"// before (client reading a truncated file)\nparquetReader.read(); // throws ParquetDecodingException\n// after\nif (!fileExistsFully(objectStats, expectedSize)) {\n    reDownloadFile();\n}\nparquetReader.read();","handlingStrategy":"try-catch","validationCode":"long rowsInChunk = columnChunkMetaData.getValueCount();\nif (batchSize > rowsInChunk) { throw new IllegalStateException(\"batch exceeds chunk values\"); }","typeGuard":null,"tryCatchPattern":"try {\n    ColumnChunk chunk = reader.readNext();\n} catch (ParquetDecodingException e) {\n    handleCorruptFile(e); // failover / re-read from backup\n}","preventionTips":["Validate Parquet files (parquet-tools) after writing and before querying","Watch for truncated objects on S3/HDFS (compare content length to footer-declared size)","Keep writer and reader versions aligned; prefer well-tested writers","Fail fast on checksum errors instead of continuing reads"],"tags":["parquet","decoding","corrupt-file"],"backgroundTag":"parquet-decode-batch-short-read","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}