prestodb/presto · error · IllegalStateException

reference to a non-existent key

Error message

reference to a non-existent key

What it means

During DictionaryBlock compaction (e.g. in getRegion/copy-with-remapping paths), each position's dictionary id is remapped through a remapIndex built from the set of dictionary positions to copy. A remapped id of -1 means the block references a dictionary entry that was not in the set of positions to keep. The library throws IllegalStateException because the resulting block would be corrupt.

Source

Thrown at presto-common/src/main/java/com/facebook/presto/common/block/DictionaryBlock.java:616

            int dictionaryIndex = getId(i);
            if (remapIndex[dictionaryIndex] == -1) {
                dictionaryPositionsToCopy.add(dictionaryIndex);
                remapIndex[dictionaryIndex] = newIndex;
                newIndex++;
            }
        }

        // entire dictionary is referenced
        if (dictionaryPositionsToCopy.size() == dictionarySize) {
            return this;
        }

        // compact the dictionary
        int[] newIds = new int[positionCount];
        for (int i = 0; i < positionCount; i++) {
            int newId = remapIndex[getId(i)];
            if (newId == -1) {
                throw new IllegalStateException("reference to a non-existent key");
            }
            newIds[i] = newId;
        }
        try {
            Block compactDictionary = dictionary.copyPositions(dictionaryPositionsToCopy.elements(), 0, dictionaryPositionsToCopy.size());
            return new DictionaryBlock(positionCount, compactDictionary, newIds, true);
        }
        catch (UnsupportedOperationException e) {
            // ignore if copy positions is not supported for the dictionary block
            return this;
        }
    }

    @Override
    public byte getByteUnchecked(int internalPosition)
    {
        assert internalPositionInRange(internalPosition, getOffsetBase(), getPositionCount());
        return dictionary.getByte(ids[internalPosition]);

View on GitHub (pinned to 55bb57d202)

Solutions

1. Verify the set of dictionary positions to copy covers every id referenced by the block before compaction (collect ids via getIds and check remapIndex has no -1 for them).
2. Regenerate the ids array from the current dictionary instead of reusing a stale one.
3. If you control construction, use DictionaryBlock.validate() / ensure the block passed validation at creation time to catch inconsistency early.

Example fix

// before
Block compacted = regionBlock.copyPositions(positions, 0, positions.length); // ids reference dropped dictionary entries
// after
int[] ids = dictionaryBlock.getIds().elements();
boolean[] keep = new boolean[dictionaryBlock.getDictionary().getPositionCount()];
for (int id : ids) { keep[id] = true; } // include ALL referenced dictionary positions in the copy set

int[] ids = dictionaryBlock.getIds().elements();
for (int id : ids) {
    if (id < 0 || id >= dictionaryBlock.getDictionary().getPositionCount()) {
        throw new IllegalStateException("id " + id + " out of dictionary range before compaction");
    }
}

boolean idsCoveredBy(int[] ids, boolean[] dictionaryPositionsKept) {
    for (int id : ids) {
        if (id < 0 || id >= dictionaryPositionsKept.length || !dictionaryPositionsKept[id]) return false;
    }
    return true;
}

try {
    compacted = block.copyPositions(...);
} catch (IllegalStateException e) {
    if (e.getMessage().contains("non-existent key")) {
        throw new corruptBlockException("dictionary compaction: block ids reference dropped dictionary entries", e);
    }
    throw e;
}

When it happens

Trigger: Compacting a DictionaryBlock whose ids array references a dictionary position not included in dictionaryPositionsToCopy; typically from a corrupted or incorrectly sliced block where remapIndex[getId(i)] == -1.

Common situations: Region extraction or copy operations where the dictionary-position selection was computed from a different (stale or shorter) id array; bugs in operators that shrink the dictionary but not the ids; deserialized blocks with inconsistent internal state.

Related errors

• reference to a non-existent key
• newDictionary must have the same position count
• dictionarySourceIds must be the same
• New column does not have same number of rows as old column
• Declared positions (%s) does not match block %s's number of

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/61ea1256e41f5936. Report an issue: GitHub.