{"record":{"id":"254c3dc8042b5ae0","repo":"apache/cassandra","slug":"corrupted-key-cache-key-length-of-d-is-longer-th","errorCode":null,"errorMessage":"Corrupted key cache. Key length of %d is longer than maximum of %d","messagePattern":"Corrupted key cache\\. Key length of (.+?) is longer than maximum of (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"src/java/org/apache/cassandra/service/CacheService.java","lineNumber":505,"sourceCode":"        {\n            AbstractRowIndexEntry entry = CacheService.instance.keyCache.getInternal(key);\n            if (entry == null)\n                return;\n\n            writeSSTable(cfs, key.desc, out);\n            out.writeInt(key.key.length);\n            out.write(key.key);\n            entry.serializeForCache(out);\n        }\n\n        public Future<Pair<KeyCacheKey, AbstractRowIndexEntry>> deserialize(DataInputPlus input) throws IOException\n        {\n            Pair<KeyCacheSupport<?>, SSTableFormat<?, ?>> reader = readSSTable(input);\n            boolean skipEntry = reader.left == null || !reader.left.getKeyCache().isEnabled();\n\n            int keyLength = input.readInt();\n            if (keyLength > FBUtilities.MAX_UNSIGNED_SHORT)\n                throw new IOException(String.format(\"Corrupted key cache. Key length of %d is longer than maximum of %d\",\n                                                    keyLength, FBUtilities.MAX_UNSIGNED_SHORT));\n            ByteBuffer key = ByteBufferUtil.read(input, keyLength);\n\n            if (skipEntry)\n            {\n                // The sstable doesn't exist anymore, so we can't be sure of the exact version and assume its the current version. The only case where we'll be\n                // wrong is during upgrade, in which case we fail at deserialization. This is not a huge deal however since 1) this is unlikely enough that\n                // this won't affect many users (if any) and only once, 2) this doesn't prevent the node from starting and 3) CASSANDRA-10219 shows that this\n                // part of the code has been broken for a while without anyone noticing (it is, btw, still broken until CASSANDRA-10219 is fixed).\n                SSTableFormat.KeyCacheValueSerializer<?, ?> serializer = reader.right.getKeyCacheValueSerializer();\n\n                serializer.skip(input);\n                return null;\n            }\n            long pos = ((RandomAccessReader) input).getPosition();\n            AbstractRowIndexEntry cacheValue;\n            try\n            {","sourceCodeStart":487,"sourceCodeEnd":523,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/service/CacheService.java#L487-L523","documentation":"While loading the saved key cache from disk, CacheService.KeyCacheSerializer.deserialize reads each entry's key length and rejects values exceeding FBUtilities.MAX_UNSIGNED_SHORT (65535) as an IOException, since such a length cannot have been written by a valid serializer run. This indicates the cache file is corrupt, truncated mid-entry, or was written by an incompatible version whose stream layout shifted.","triggerScenarios":"Node startup cache load reading a key cache file where the int read as keyLength exceeds 65535 — i.e. misaligned reads after truncation, disk corruption, or deserializing a file produced by an incompatible Cassandra version.","commonSituations":"Unclean shutdown (OOM/kill -9) that truncated the cache file; disk-level corruption; restore of cache files from a mismatched backup; upgrades across versions with changed cache serialization layout.","solutions":["Delete the corrupted key cache file (saved_caches directory, e.g. /var/lib/cassandra/saved_caches/*) and restart — Cassandra rebuilds the cache.","Clear the whole saved_caches directory if multiple cache files fail to load.","Check filesystem/disk health (dmesg, fsck) if corruption recurs.","After version upgrades, do not carry over cache files from the previous version's data directory."],"exampleFix":"// before\nsystemctl start cassandra   # fails/loops on corrupted key cache\n// after\nrm -f /var/lib/cassandra/saved_caches/*\nsystemctl start cassandra   # cache rebuilt from SSTables","handlingStrategy":"try-catch","validationCode":"// Pre-start check: file size must plausibly hold valid entries (not truncated)\nFile f = new File(savedCachesDir, \"KeyCache-*.db\");\nif (f.exists() && f.length() < 8) Files.delete(f.toPath()); // too small to be valid, delete and let it rebuild","typeGuard":null,"tryCatchPattern":"try {\n    cacheService.loadCaches();\n} catch (IOException e) {\n    if (e.getMessage().startsWith(\"Corrupted key cache\")) {\n        logger.warn(\"Key cache file corrupted, deleting and rebuilding\", e);\n        FileUtils.deleteQuietly(new File(savedCachesDir, \"KeyCache-*.db\"));\n        cacheService.loadCaches(); // retry after cleanup\n    } else {\n        throw e;\n    }\n}","preventionTips":["Shut down nodes cleanly; avoid kill -9 / OOM kills that truncate cache files.","Do not restore saved_caches from backups of different versions or other nodes.","Monitor disk health; recurring corruption points at hardware issues.","After Cassandra upgrades, clear saved_caches before first start.","Keep cache load failures alertable so startup is not silently degraded."],"tags":["cache","deserialization","corruption","startup"],"backgroundTag":"checksum-mismatch","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}