prestodb/presto · error · java.lang.IllegalStateException

reference to a non-existent key

Error message

reference to a non-existent key

What it means

getNewIds remaps each dictionary id in a DictionaryBlock through remapIndex built from the positions of the dictionary that will be retained. A remap value of -1 means the block references a dictionary entry that was not marked for retention, so the compacted dictionary would be missing an entry. The library throws IllegalStateException instead of producing a corrupt page.

Source

Thrown at presto-common/src/main/java/com/facebook/presto/common/Page.java:312

            try {
                Block compactDictionary = dictionaryBlock.getDictionary().copyPositions(dictionaryPositionsToCopy, 0, numberOfIndexes);
                outputDictionaryBlocks.add(new DictionaryBlock(positionCount, compactDictionary, newIds, true, newDictionaryId));
            }
            catch (UnsupportedOperationException e) {
                // ignore if copy positions is not supported for the dictionary
                outputDictionaryBlocks.add(dictionaryBlock);
            }
        }
        return outputDictionaryBlocks;
    }

    private static int[] getNewIds(int positionCount, DictionaryBlock dictionaryBlock, int[] remapIndex)
    {
        int[] newIds = new int[positionCount];
        for (int i = 0; i < positionCount; i++) {
            int newId = remapIndex[dictionaryBlock.getId(i)];
            if (newId == -1) {
                throw new IllegalStateException("reference to a non-existent key");
            }
            newIds[i] = newId;
        }
        return newIds;
    }

    /**
     * Returns a page that assures all data is in memory.
     * May return the same page if all page data is already in memory.
     * <p>
     * This allows streaming data sources to skip sections that are not
     * accessed in a query.
     */
    public Page getLoadedPage()
    {
        for (int i = 0; i < blocks.length; i++) {
            Block loaded = blocks[i].getLoadedBlock();
            if (loaded != blocks[i]) {

View on GitHub (pinned to 55bb57d202)

Solutions

1. Verify all blocks with equal DictionarySourceId truly share identical dictionary contents; fix the producer that fabricates/collides dictionary ids
2. Decode the DictionaryBlock into a plain block (getLoadedBlock/copyPositions) before compaction
3. Rebuild the Page from source data so dictionary and ids are regenerated consistently
4. Check for Presto version/serialization issues that duplicate dictionary ids and upgrade

Example fix

// before
// block claims same source id but has extra ids -> remapIndex[id] == -1
page.compactBlocks();
// after
Block decoded = dictionaryBlock.getLoadedBlock(); // or copy to plain block
Page fixed = new Page(decoded, otherBlocks);
fixed.compactBlocks();

int[] remap = buildRemapIndex(dictionaryPositionsToCopy);
for (int i = 0; i < dictionaryBlock.getPositionCount(); i++) {
    if (remap[dictionaryBlock.getId(i)] == -1) {
        throw new IllegalStateException("dictionary id " + dictionaryBlock.getId(i) + " not retained; decode block first");
    }
}

boolean idsWithinRetained(DictionaryBlock block, boolean[] retained) {
    for (int i = 0; i < block.getPositionCount(); i++) {
        if (!retained[block.getId(i)]) return false;
    }
    return true;
}

try {
    return compactDictionaryPage(page);
} catch (IllegalStateException e) {
    if (!e.getMessage().contains("non-existent key")) throw e;
    return decodeToPlainBlocks(page); // fallback path
}

When it happens

Trigger: Page.compactBlocks where a DictionaryBlock's ids point at dictionary positions that dictionaryPositionsToCopy did not include — typically when blocks claim the same DictionarySourceId but actually have divergent dictionary contents, or remapIndex was computed against a different dictionary than the block's.

Common situations: Bugs in custom Block implementations that lie about their DictionarySourceId; dictionary id collisions from randomDictionaryId after deserializing pages across versions.

Related errors

• dictionarySourceIds must be the same
• reference to a non-existent key
• newDictionary must have the same position count
• field %s has unexpected position count. Expected: %s, actual
• INVALID_TABLE_PROPERTY

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/573adb17045adf18. Report an issue: GitHub.