apache/cassandra · error · IOException

Corrupted key cache. Failed to deserialize key of key cache

Error message

Corrupted key cache. Failed to deserialize key of key cache - invalid sstable ordinal ${ordinal}

What it means

KeyCacheSerializer.readSSTable reads an unsigned VInt ordinal indexing the per-deserialization list of live SSTable readers; an ordinal beyond readers.size() cannot correspond to any existing SSTable, so it throws an IOException declaring the key cache data corrupted. The reader list is built at deserialize time, so a stale or mismatched cache file yields out-of-range ordinals.

Source

Thrown at src/java/org/apache/cassandra/service/CacheService.java:546

                throw ex;
            }
            KeyCacheKey cacheKey = reader.left.getCacheKey(key);
            return ImmediateFuture.success(Pair.create(cacheKey, cacheValue));
        }

        private void writeSSTable(ColumnFamilyStore cfs, Descriptor desc, DataOutputPlus out) throws IOException
        {
            getOrCreateCFSOrdinal(cfs);
            Pair<Integer, ColumnFamilyStore> existing = readerOrdinals.putIfAbsent(desc, Pair.create(readerOrdinals.size(), cfs));
            int ordinal = existing == null ? readerOrdinals.size() - 1 : existing.left;
            out.writeUnsignedVInt32(ordinal);
        }

        private Pair<KeyCacheSupport<?>, SSTableFormat<?, ?>> readSSTable(DataInputPlus input) throws IOException
        {
            int ordinal = input.readUnsignedVInt32();
            if (ordinal >= readers.size())
                throw new IOException("Corrupted key cache. Failed to deserialize key of key cache - invalid sstable ordinal " + ordinal);
            return readers.get(ordinal);
        }

        public void cleanupAfterDeserialize()
        {
            super.cleanupAfterDeserialize();
            readers.clear();
        }

        public void cleanupAfterSerialize()
        {
            super.cleanupAfterSerialize();
            readerOrdinals.clear();
        }
    }
}

View on GitHub (pinned to 88fd0f6a0e)

Solutions

Remove the key cache file(s) from the saved_caches directory and restart so the cache is rebuilt.
Never copy saved_caches between nodes or across Cassandra versions; let each node repopulate its cache.
Verify SSTable data directory integrity if the reader list unexpectedly shrank (missing SSTables).
Check disk health if the corruption reproduces after cache regeneration.

Example fix

// before
scp nodeA:/var/lib/cassandra/saved_caches/* nodeB:/var/lib/cassandra/saved_caches/  # then start nodeB
// after
rm -f /var/lib/cassandra/saved_caches/*  # rebuild cache locally on nodeB
systemctl start cassandra

Defensive patterns

Strategy: try-catch

Validate before calling

// Validate cache file belongs to this node/version before load
File f = new File(savedCachesDir, "KeyCache-*.db");
if (!CacheService.validateCacheFileForCurrentSSTables(f)) Files.delete(f.toPath());

Try / catch

try {
    cacheService.loadCaches();
} catch (IOException e) {
    if (e.getMessage().contains("invalid sstable ordinal")) {
        logger.warn("Stale/corrupted key cache file, discarding", e);
        FileUtils.cleanDirectory(savedCachesDir);
        // cache will be rebuilt from SSTables on next access
    } else {
        throw e;
    }
}

Prevention

Never copy saved_caches directories between nodes or clusters.
Clear saved_caches after version upgrades before starting the node.
Ensure SSTables are not deleted/moved independently of cache state (use nodetool cleanup instead of rm).
Check filesystem integrity when ordinal errors appear repeatedly.
Keep cache save/load on the same node lifecycle to avoid stale SSTable references.

When it happens

Trigger: Loading a saved key cache whose entries reference SSTable indices that exceed the number of SSTables registered at load time — typically after truncation/corruption, or when the cache file was written by a different set of SSTables/version than the ones present at startup.

Common situations: Restoring saved_caches from another node or backup; upgrading Cassandra and reusing old cache files; partially deleted SSTables combined with a still-populated key cache file; disk corruption shifting VInt boundaries.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/2523ee7a2dbed679. Report an issue: GitHub.