prestodb/presto · error · OrcCorruptionException

Value is not null but data stream is not present

Error message

Value is not null but data stream is not present

What it means

In MapDirectBatchStreamReader.readBlock, when a row's map value is non-null the reader must have a length stream to locate that entry's key/value span. If the row is marked present but the length (data) stream is absent, the file's stream metadata is inconsistent with its data, so the reader throws OrcCorruptionException('Value is not null but data stream is not present'). This guards against reading a physically incomplete column.

Source

Thrown at presto-orc/src/main/java/com/facebook/presto/orc/reader/MapDirectBatchStreamReader.java:114

    }

    @Override
    public Block readBlock()
            throws IOException
    {
        if (!rowGroupOpen) {
            openRowGroup();
        }

        if (readOffset > 0) {
            if (presentStream != null) {
                // skip ahead the present bit reader, but count the set bits
                // and use this as the skip size for the data reader
                readOffset = presentStream.countBitsSet(readOffset);
            }
            if (readOffset > 0) {
                if (lengthStream == null) {
                    throw new OrcCorruptionException(streamDescriptor.getOrcDataSourceId(), "Value is not null but data stream is not present");
                }
                long entrySkipSize = lengthStream.sum(readOffset);
                keyStreamReader.prepareNextRead(toIntExact(entrySkipSize));
                valueStreamReader.prepareNextRead(toIntExact(entrySkipSize));
            }
        }

        // We will use the offsetVector as the buffer to read the length values from lengthStream,
        // and the length values will be converted in-place to an offset vector.
        int[] offsetVector = new int[nextBatchSize + 1];
        boolean[] nullVector = null;

        if (presentStream == null) {
            if (lengthStream == null) {
                throw new OrcCorruptionException(streamDescriptor.getOrcDataSourceId(), "Value is not null but data stream is not present");
            }
            lengthStream.next(offsetVector, nextBatchSize);
        }

View on GitHub (pinned to 55bb57d202)

Solutions

1. Validate the file with orc-tools (or a full scan of the column) to confirm the length stream is missing in the footer/metadata.
2. Regenerate or re-write the file with a known-good writer; replace the corrupted copy from a backup.
3. Check the writer version that produced the file for known bugs emitting maps without length streams.
4. If the corruption is expected, filter/repartition around the bad file and exclude it from the table.

Example fix

// before — trusting the file blindly
reader.readBlock(mapType, positions);
// after — pre-validate with a checksum/validation pass
boolean ok = new OrcValidator(orcDataSource).validate();
if (!ok) { skipFile(); } else { reader.readBlock(mapType, positions); }

// before reading, verify the column has the required streams
List<StreamId> streams = getStreamsForColumn(columnId);
boolean hasLength = streams.stream().anyMatch(s -> s.getStreamKind() == StreamKind.LENGTH);
boolean hasPresent = streams.stream().anyMatch(s -> s.getStreamKind() == StreamKind.PRESENT);
if (!hasLength && !hasPresent) {
    throw new PrestoException(ORC_BAD_DATA, "Map column missing LENGTH stream in stripe");
}

try {
    block = mapReader.readBlock(mapType, positions);
}
catch (OrcCorruptionException e) {
    throw new PrestoException(ORC_BAD_DATA, "Corrupted map column (missing length stream): " + e.getMessage(), e);
}

When it happens

Trigger: Reading a stripe where the present stream says at least one row is non-null but no length stream exists for the map column, and readOffset > 0 requires skipping entries via lengthStream.sum(readOffset).

Common situations: Truncated or partially-written ORC/DWRF files (writer crashed mid-stripe); files produced by buggy third-party writers omitting the length stream; manual file splicing/merging that dropped streams; wrong column-to-stream mapping after schema evolution.

Related errors

• Invalid map block:
• Type must be a MapType for MapVector
• INVALID_FUNCTION_ARGUMENT
• NOT_SUPPORTED
• NOT_SUPPORTED

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/ad64a47e3803aa6e. Report an issue: GitHub.