{"record":{"id":"52f8725fe392f474","repo":"prestodb/presto","slug":"value-is-not-null-but-data-stream-is-not-present-52f872","errorCode":null,"errorMessage":"Value is not null but data stream is not present","messagePattern":"Value is not null but data stream is not present","errorType":"exception","errorClass":"OrcCorruptionException","httpStatus":null,"severity":"error","filePath":"presto-orc/src/main/java/com/facebook/presto/orc/reader/ListBatchStreamReader.java","lineNumber":106,"sourceCode":"    }\n\n    @Override\n    public Block readBlock()\n            throws IOException\n    {\n        if (!rowGroupOpen) {\n            openRowGroup();\n        }\n\n        if (readOffset > 0) {\n            if (presentStream != null) {\n                // skip ahead the present bit reader, but count the set bits\n                // and use this as the skip size for the data reader\n                readOffset = presentStream.countBitsSet(readOffset);\n            }\n            if (readOffset > 0) {\n                if (lengthStream == null) {\n                    throw new OrcCorruptionException(streamDescriptor.getOrcDataSourceId(), \"Value is not null but data stream is not present\");\n                }\n                long elementSkipSize = lengthStream.sum(readOffset);\n                elementStreamReader.prepareNextRead(toIntExact(elementSkipSize));\n            }\n        }\n\n        // We will use the offsetVector as the buffer to read the length values from lengthStream,\n        // and the length values will be converted in-place to an offset vector.\n        int[] offsetVector = new int[nextBatchSize + 1];\n        boolean[] nullVector = null;\n        if (presentStream == null) {\n            if (lengthStream == null) {\n                throw new OrcCorruptionException(streamDescriptor.getOrcDataSourceId(), \"Value is not null but data stream is not present\");\n            }\n            lengthStream.next(offsetVector, nextBatchSize);\n        }\n        else {\n            nullVector = new boolean[nextBatchSize];","sourceCodeStart":88,"sourceCodeEnd":124,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-orc/src/main/java/com/facebook/presto/orc/reader/ListBatchStreamReader.java#L88-L124","documentation":"ListBatchStreamReader.readBlock throws this (message text says 'data stream' but the guard is on lengthStream) when seeking past readOffset values: the LIST column's LENGTH stream is needed to compute the total element skip size, but it is absent. The stream carries the element counts of each list value, so without it the reader cannot know how many child elements to skip — the file is treated as corrupt.","triggerScenarios":"readBlock seek path with readOffset > 0 where the stripe has no LENGTH stream for the LIST column (lengthStream == null).","commonSituations":"Corrupt ORC files missing the list LENGTH stream; buggy writers writing offsets/lengths incorrectly; schema/stream mismatch after partial rewrite; incompatible writer versions.","solutions":["Dump stripe metadata with orc-tools to confirm the LENGTH stream is missing for the list column.","Rewrite the file with a compliant ORC writer and re-read.","Fix the upstream writer to always emit the LENGTH stream for list columns.","Validate files at ingestion (stripe stream checks) before analytics use."],"exampleFix":"// before\n// reading a corrupted file directly\ntry { pageSource.getNextPage(); }\n// after\ncatch (OrcCorruptionException e) {\n    metrics.corruptFileCounter.increment();\n    reprocessFromSource(file, e);\n}","handlingStrategy":"try-catch","validationCode":"// LIST columns always require a LENGTH stream\nif (!stripeHasStream(file, listColumnId, StreamKind.LENGTH)) {\n    throw new IllegalStateException(\"LIST column missing LENGTH stream in \" + file);\n}","typeGuard":null,"tryCatchPattern":"try {\n    pageSource.getNextPage();\n} catch (OrcCorruptionException e) {\n    log.error(\"List LENGTH stream missing; file corrupt: %s\", e.getMessage());\n    throw e;\n}","preventionTips":["Verify list-column stream layout with orc-tools meta for new producers.","Rewrite suspicious files instead of partial reads.","Cross-check present-stream null counts against available streams at ingest.","Fix writers that drop LENGTH streams for empty or null-heavy row groups."],"tags":["orc","corruption","list","missing-stream"],"backgroundTag":"orc-corrupt-file","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}