prestodb/presto · error · OrcCorruptionException

Dictionary is not empty but dictionary length stream is not

Error message

Dictionary is not empty but dictionary length stream is not present

What it means

During openRowGroup, the reader found stripeDictionarySize > 0 (the stripe dictionary holds entries) but the DICTIONARY_LENGTH stream is absent, so entry lengths cannot be decoded. The file's stream layout contradicts its metadata, so it is flagged as corrupt.

Source

Thrown at presto-orc/src/main/java/com/facebook/presto/orc/reader/SliceDictionaryBatchStreamReader.java:228

        }
    }

    private void openRowGroup()
            throws IOException
    {
        // read the dictionary
        if (!stripeDictionaryOpen) {
            if (stripeDictionarySize > 0) {
                // resize the dictionary lengths array if necessary
                if (stripeDictionaryLength.length < stripeDictionarySize) {
                    stripeDictionaryLength = new int[stripeDictionarySize];
                    systemMemoryContext.setBytes(sizeOf(stripeDictionaryLength));
                }

                // read the lengths
                LongInputStream lengthStream = stripeDictionaryLengthStreamSource.openStream();
                if (lengthStream == null) {
                    throw new OrcCorruptionException(streamDescriptor.getOrcDataSourceId(), "Dictionary is not empty but dictionary length stream is not present");
                }
                lengthStream.next(stripeDictionaryLength, stripeDictionarySize);

                long dataLength = 0;
                for (int i = 0; i < stripeDictionarySize; i++) {
                    dataLength += stripeDictionaryLength[i];
                }

                // we must always create a new dictionary array because the previous dictionary may still be referenced
                stripeDictionaryData = new byte[toIntExact(dataLength)];
                systemMemoryContext.setBytes(sizeOf(stripeDictionaryData));

                // add one extra entry for null
                stripeDictionaryOffsetVector = new int[stripeDictionarySize + 2];
                systemMemoryContext.setBytes(sizeOf(stripeDictionaryOffsetVector));

                // read dictionary values
                ByteArrayInputStream dictionaryDataStream = stripeDictionaryDataStreamSource.openStream();

View on GitHub (pinned to 55bb57d202)

Solutions

1. Validate the ORC file with orc-tools to confirm the missing DICTIONARY_LENGTH stream.
2. Regenerate the file or restore from backup.
3. Ensure no concurrent writers overwrite the file while it is being read (write-then-rename pattern).
4. Check the producing writer version for known dictionary-encoding bugs.

// pre-flight: confirm DICTIONARY_LENGTH stream exists when dictionary encoding is used
// orc-tools meta file.orc | grep DICTIONARY_LENGTH

try {
    return reader.readBlock();
} catch (OrcCorruptionException e) {
    log.error("Dictionary length stream missing in %s", e.getOrcDataSourceId());
    throw new DataIntegrityException("ORC dictionary encoding incomplete", e);
}

When it happens

Trigger: readBlock() triggering openRowGroup() on a dictionary-encoded string column with stripeDictionarySize > 0 while stripeDictionaryLengthStreamSource.openStream() returns null.

Common situations: Truncated ORC files missing tail streams; files written by buggy writers that emit dictionary data without lengths; corruption during file transfer or partial overwrite by concurrent writers.

Related errors

• Value is not null but data stream is not present
• Dictionary is not empty but dictionary length stream is not
• Value is not null but data stream is not present
• Value is not null but data stream is missing
• Value is not null but decimal stream is not present

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/2ebe08fc6737368b. Report an issue: GitHub.