{"record":{"id":"2523ee7a2dbed679","repo":"apache/cassandra","slug":"corrupted-key-cache-failed-to-deserialize-key-of","errorCode":null,"errorMessage":"Corrupted key cache. Failed to deserialize key of key cache - invalid sstable ordinal ${ordinal}","messagePattern":"Corrupted key cache\\. Failed to deserialize key of key cache - invalid sstable ordinal (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"src/java/org/apache/cassandra/service/CacheService.java","lineNumber":546,"sourceCode":"                throw ex;\n            }\n            KeyCacheKey cacheKey = reader.left.getCacheKey(key);\n            return ImmediateFuture.success(Pair.create(cacheKey, cacheValue));\n        }\n\n        private void writeSSTable(ColumnFamilyStore cfs, Descriptor desc, DataOutputPlus out) throws IOException\n        {\n            getOrCreateCFSOrdinal(cfs);\n            Pair<Integer, ColumnFamilyStore> existing = readerOrdinals.putIfAbsent(desc, Pair.create(readerOrdinals.size(), cfs));\n            int ordinal = existing == null ? readerOrdinals.size() - 1 : existing.left;\n            out.writeUnsignedVInt32(ordinal);\n        }\n\n        private Pair<KeyCacheSupport<?>, SSTableFormat<?, ?>> readSSTable(DataInputPlus input) throws IOException\n        {\n            int ordinal = input.readUnsignedVInt32();\n            if (ordinal >= readers.size())\n                throw new IOException(\"Corrupted key cache. Failed to deserialize key of key cache - invalid sstable ordinal \" + ordinal);\n            return readers.get(ordinal);\n        }\n\n        public void cleanupAfterDeserialize()\n        {\n            super.cleanupAfterDeserialize();\n            readers.clear();\n        }\n\n        public void cleanupAfterSerialize()\n        {\n            super.cleanupAfterSerialize();\n            readerOrdinals.clear();\n        }\n    }\n}\n","sourceCodeStart":528,"sourceCodeEnd":563,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/service/CacheService.java#L528-L563","documentation":"KeyCacheSerializer.readSSTable reads an unsigned VInt ordinal indexing the per-deserialization list of live SSTable readers; an ordinal beyond readers.size() cannot correspond to any existing SSTable, so it throws an IOException declaring the key cache data corrupted. The reader list is built at deserialize time, so a stale or mismatched cache file yields out-of-range ordinals.","triggerScenarios":"Loading a saved key cache whose entries reference SSTable indices that exceed the number of SSTables registered at load time — typically after truncation/corruption, or when the cache file was written by a different set of SSTables/version than the ones present at startup.","commonSituations":"Restoring saved_caches from another node or backup; upgrading Cassandra and reusing old cache files; partially deleted SSTables combined with a still-populated key cache file; disk corruption shifting VInt boundaries.","solutions":["Remove the key cache file(s) from the saved_caches directory and restart so the cache is rebuilt.","Never copy saved_caches between nodes or across Cassandra versions; let each node repopulate its cache.","Verify SSTable data directory integrity if the reader list unexpectedly shrank (missing SSTables).","Check disk health if the corruption reproduces after cache regeneration."],"exampleFix":"// before\nscp nodeA:/var/lib/cassandra/saved_caches/* nodeB:/var/lib/cassandra/saved_caches/  # then start nodeB\n// after\nrm -f /var/lib/cassandra/saved_caches/*  # rebuild cache locally on nodeB\nsystemctl start cassandra","handlingStrategy":"try-catch","validationCode":"// Validate cache file belongs to this node/version before load\nFile f = new File(savedCachesDir, \"KeyCache-*.db\");\nif (!CacheService.validateCacheFileForCurrentSSTables(f)) Files.delete(f.toPath());","typeGuard":null,"tryCatchPattern":"try {\n    cacheService.loadCaches();\n} catch (IOException e) {\n    if (e.getMessage().contains(\"invalid sstable ordinal\")) {\n        logger.warn(\"Stale/corrupted key cache file, discarding\", e);\n        FileUtils.cleanDirectory(savedCachesDir);\n        // cache will be rebuilt from SSTables on next access\n    } else {\n        throw e;\n    }\n}","preventionTips":["Never copy saved_caches directories between nodes or clusters.","Clear saved_caches after version upgrades before starting the node.","Ensure SSTables are not deleted/moved independently of cache state (use nodetool cleanup instead of rm).","Check filesystem integrity when ordinal errors appear repeatedly.","Keep cache save/load on the same node lifecycle to avoid stale SSTable references."],"tags":["cache","deserialization","corruption","sstable","startup"],"backgroundTag":"checksum-mismatch","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}