{"record":{"id":"61ea1256e41f5936","repo":"prestodb/presto","slug":"reference-to-a-non-existent-key-61ea12","errorCode":null,"errorMessage":"reference to a non-existent key","messagePattern":"reference to a non-existent key","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"presto-common/src/main/java/com/facebook/presto/common/block/DictionaryBlock.java","lineNumber":616,"sourceCode":"            int dictionaryIndex = getId(i);\n            if (remapIndex[dictionaryIndex] == -1) {\n                dictionaryPositionsToCopy.add(dictionaryIndex);\n                remapIndex[dictionaryIndex] = newIndex;\n                newIndex++;\n            }\n        }\n\n        // entire dictionary is referenced\n        if (dictionaryPositionsToCopy.size() == dictionarySize) {\n            return this;\n        }\n\n        // compact the dictionary\n        int[] newIds = new int[positionCount];\n        for (int i = 0; i < positionCount; i++) {\n            int newId = remapIndex[getId(i)];\n            if (newId == -1) {\n                throw new IllegalStateException(\"reference to a non-existent key\");\n            }\n            newIds[i] = newId;\n        }\n        try {\n            Block compactDictionary = dictionary.copyPositions(dictionaryPositionsToCopy.elements(), 0, dictionaryPositionsToCopy.size());\n            return new DictionaryBlock(positionCount, compactDictionary, newIds, true);\n        }\n        catch (UnsupportedOperationException e) {\n            // ignore if copy positions is not supported for the dictionary block\n            return this;\n        }\n    }\n\n    @Override\n    public byte getByteUnchecked(int internalPosition)\n    {\n        assert internalPositionInRange(internalPosition, getOffsetBase(), getPositionCount());\n        return dictionary.getByte(ids[internalPosition]);","sourceCodeStart":598,"sourceCodeEnd":634,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-common/src/main/java/com/facebook/presto/common/block/DictionaryBlock.java#L598-L634","documentation":"During DictionaryBlock compaction (e.g. in getRegion/copy-with-remapping paths), each position's dictionary id is remapped through a remapIndex built from the set of dictionary positions to copy. A remapped id of -1 means the block references a dictionary entry that was not in the set of positions to keep. The library throws IllegalStateException because the resulting block would be corrupt.","triggerScenarios":"Compacting a DictionaryBlock whose ids array references a dictionary position not included in dictionaryPositionsToCopy; typically from a corrupted or incorrectly sliced block where remapIndex[getId(i)] == -1.","commonSituations":"Region extraction or copy operations where the dictionary-position selection was computed from a different (stale or shorter) id array; bugs in operators that shrink the dictionary but not the ids; deserialized blocks with inconsistent internal state.","solutions":["Verify the set of dictionary positions to copy covers every id referenced by the block before compaction (collect ids via getIds and check remapIndex has no -1 for them).","Regenerate the ids array from the current dictionary instead of reusing a stale one.","If you control construction, use DictionaryBlock.validate() / ensure the block passed validation at creation time to catch inconsistency early."],"exampleFix":"// before\nBlock compacted = regionBlock.copyPositions(positions, 0, positions.length); // ids reference dropped dictionary entries\n// after\nint[] ids = dictionaryBlock.getIds().elements();\nboolean[] keep = new boolean[dictionaryBlock.getDictionary().getPositionCount()];\nfor (int id : ids) { keep[id] = true; } // include ALL referenced dictionary positions in the copy set","handlingStrategy":"validation","validationCode":"int[] ids = dictionaryBlock.getIds().elements();\nfor (int id : ids) {\n    if (id < 0 || id >= dictionaryBlock.getDictionary().getPositionCount()) {\n        throw new IllegalStateException(\"id \" + id + \" out of dictionary range before compaction\");\n    }\n}","typeGuard":"boolean idsCoveredBy(int[] ids, boolean[] dictionaryPositionsKept) {\n    for (int id : ids) {\n        if (id < 0 || id >= dictionaryPositionsKept.length || !dictionaryPositionsKept[id]) return false;\n    }\n    return true;\n}","tryCatchPattern":"try {\n    compacted = block.copyPositions(...);\n} catch (IllegalStateException e) {\n    if (e.getMessage().contains(\"non-existent key\")) {\n        throw new corruptBlockException(\"dictionary compaction: block ids reference dropped dictionary entries\", e);\n    }\n    throw e;\n}","preventionTips":["Build the keep-set of dictionary positions from the block's actual ids, never from a cached list.","Run DictionaryBlock.validate on blocks crossing operator boundaries in tests.","Treat this error as data corruption: fail the query rather than retrying."],"tags":["presto","block","dictionary-encoding","corrupt-state"],"backgroundTag":"dictionary-compaction-missing-key","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}