apache/cassandra · warning

Error loading counter cache

Error message

Error loading counter cache

What it means

WARN from StorageService.initServer() when loading the saved counter cache (CacheService.instance.counterCache.loadSavedAsync().get()) fails. The original Throwable is passed to JVMStabilityInspector.inspectThrowable to decide whether it is fatal (e.g. disk corruption may halt the JVM), and if execution continues the node starts without a warmed counter cache — counters still work but the cache rebuilds cold.

Source

Thrown at src/java/org/apache/cassandra/service/StorageService.java:849

                                    ClusterMetadataService.state() != ClusterMetadataService.State.GOSSIP); // only populate local state if not running in gossip mode
        }

        Gossiper.instance.register(this);
        Gossiper.instance.addLocalApplicationState(ApplicationState.NET_VERSION, valueFactory.networkVersion());
        Gossiper.instance.addLocalApplicationState(ApplicationState.SSTABLE_VERSIONS,
                                                   valueFactory.sstableVersions(sstablesTracker.versionsInUse()));

        Gossiper.instance.triggerRoundWithCMS();

        // Has to be called after the host id has potentially changed
        try
        {
            CacheService.instance.counterCache.loadSavedAsync().get();
        }
        catch (Throwable t)
        {
            JVMStabilityInspector.inspectThrowable(t);
            logger.warn("Error loading counter cache", t);
        }
        Gossiper.waitToSettle();

        NodeId self = Register.maybeRegister();
        if (!AccordService.isSetupOrStarting())
            AccordService.localStartup(self);
        AccordService.distributedStartup();

        RegistrationStatus.instance.onRegistration();
        Startup.maybeExecuteStartupTransformation(self);

        if (CassandraRelevantProperties.SYNC_SYSTEM_PEERS_TABLES_AT_STARTUP.getBoolean())
            SystemPeersValidator.validateAndRepair(ClusterMetadata.current());

        try
        {
            if (joinRing)
                joinRing();

View on GitHub (pinned to 88fd0f6a0e)

Solutions

1. Read the attached stack trace; if JVMStabilityInspector flagged disk errors, address disk health first.
2. Delete the counter cache files under the saved_caches directory (safe — cache rebuilds automatically) and restart.
3. Check directory permissions/ownership for the cassandra user on saved_caches.
4. Verify disk space and filesystem integrity (dmesg, fsck) if corruption repeats.
5. Optionally disable the counter cache (counter_cache_save_period: 0) if counters are not used.

Example fix

# before: corrupted cache blocks startup behavior
ls /var/lib/cassandra/saved_caches/
# after: remove stale cache files and restart
rm /var/lib/cassandra/saved_caches/CounterCache-*
systemctl restart cassandra

// preflight: check saved_caches dir readable and not corrupted
File dir = new File(DatabaseDescriptor.getSavedCachesLocation());
if (!dir.canRead()) logger.warn("saved_caches unreadable: {}", dir);

try {
    CacheService.instance.counterCache.loadSavedAsync().get();
} catch (Throwable t) {
    JVMStabilityInspector.inspectThrowable(t);
    logger.warn("Error loading counter cache", t);
}

When it happens

Trigger: initServer() startup path where counter cache files in saved_caches are unreadable, corrupt, or an IO/runtime exception occurs during async load; .get() rethrows the loading failure.

Common situations: Corrupted or truncated saved_caches files after an unclean shutdown or disk failure; permission problems on the saved_caches directory; restoring a node from backup without cache files.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Error loading key or row cache
• Cache schema version + expected + does not match current sch
• Chunk cache size cannot be changed.
• Configured ${configName} "${intf}" could not be found
• Configured ${configName} "${intf}" was found, but had no add

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/0140c9ab6531f9e0. Report an issue: GitHub.