{"record":{"id":"e99ead19bb257b89","repo":"prestodb/presto","slug":"still-remaining-to-be-read-in-current-batch-e99ead","errorCode":null,"errorMessage":"Still remaining to be read in current batch.","messagePattern":"Still remaining to be read in current batch\\.","errorType":"exception","errorClass":"ParquetDecodingException","httpStatus":null,"severity":"error","filePath":"presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/Int32FlatBatchReader.java","lineNumber":187,"sourceCode":"                int valueDestinationIndex = startOffset + chunkSize - 1;\n                int valueSourceIndex = startOffset + nonNullCount - 1;\n\n                while (valueDestinationIndex >= startOffset) {\n                    if (!isNull[valueDestinationIndex]) {\n                        values[valueDestinationIndex] = values[valueSourceIndex];\n                        valueSourceIndex--;\n                    }\n                    valueDestinationIndex--;\n                }\n            }\n\n            startOffset += chunkSize;\n            remainingInBatch -= chunkSize;\n            remainingCountInPage -= chunkSize;\n        }\n\n        if (remainingInBatch != 0) {\n            throw new ParquetDecodingException(\"Still remaining to be read in current batch.\");\n        }\n\n        if (totalNonNullCount == 0) {\n            Block block = RunLengthEncodedBlock.create(field.getType(), null, nextBatchSize);\n            return new ColumnChunk(block, new int[0], new int[0]);\n        }\n\n        boolean hasNoNull = totalNonNullCount == nextBatchSize;\n        Block block = new IntArrayBlock(nextBatchSize, hasNoNull ? Optional.empty() : Optional.of(isNull), values);\n        return new ColumnChunk(block, new int[0], new int[0]);\n    }\n\n    private ColumnChunk readWithoutNull()\n            throws IOException\n    {\n        int[] values = new int[nextBatchSize];\n        int remainingInBatch = nextBatchSize;\n        int startOffset = 0;","sourceCodeStart":169,"sourceCodeEnd":205,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-parquet/src/main/java/com/facebook/presto/parquet/batchreader/Int32FlatBatchReader.java#L169-L205","documentation":"Int32FlatBatchReader.readWithNull throws this when, after exhausting the column chunk's pages, the requested batch of nextBatchSize int32 values could not be fully filled. It indicates page value counts (and nullability data) are inconsistent with the requested row count — the metadata promises more values than the pages contain.","triggerScenarios":"readNext on an optional int32 column where the sum of page valueCounts is less than nextBatchSize; readNextPage() returns null before remainingInBatch reaches 0.","commonSituations":"Truncated Parquet files; writers emitting wrong page valueCount metadata; corrupted dictionary or RLE bit-packed data causing premature page exhaustion.","solutions":["Validate/regenerate the Parquet file; check with parquet-tools whether the column chunk is complete","Fix or upgrade the writer that produced wrong page metadata","Upgrade the Presto reader version for better corruption diagnostics","Restore a complete copy of the file from object storage checkpoints"],"exampleFix":"// before\nrows = reader.read(batchSize);\n// after\nlong availableRows = columnChunkRowCount - rowsRead;\nrows = reader.read((int) Math.min(batchSize, availableRows));","handlingStrategy":"try-catch","validationCode":"int readable = (int) Math.min(batchSize, columnChunkMetaData.getValueCount() - rowsRead);\nif (readable < batchSize) { requestNewSplit(); }","typeGuard":null,"tryCatchPattern":"try {\n    reader.readNext();\n} catch (ParquetDecodingException e) {\n    markSplitFailedAndSkip();\n}","preventionTips":["Validate files before ingestion (parquet-tools)","Compare declared row counts to actual decoded rows periodically","Restore truncated files from source checkpoints","Keep reader/writer versions compatible"],"tags":["parquet","decoding","corrupt-file"],"backgroundTag":"parquet-decode-batch-short-read","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}