CacheStore cannot be used on the worker node! cache_keys was

Error message

CacheStore cannot be used on the worker node! cache_keys was used.

What it means

ClusterCacheStoreClient (worker-node CacheStore) panics on cache_keys; listing cache keys is a master-owned operation since only the master's rocksdb cache holds the actual items. Workers panic to prevent empty/misleading listings.

Source

Thrown at rust/cubestore/cubestore/src/cachestore/cache_rocksstore.rs:1973

    async fn cache_clear(&self) -> Result<(), CubeError> {
        panic!("CacheStore cannot be used on the worker node! cache_clear was used.")
    }

    async fn truncate(&self) -> Result<(), CubeError> {
        panic!("CacheStore cannot be used on the worker node! truncate was used.")
    }

    async fn cache_delete(&self, _key: String) -> Result<(), CubeError> {
        panic!("CacheStore cannot be used on the worker node! cache_delete was used.")
    }

    async fn cache_get(&self, _key: String) -> Result<Option<IdRow<CacheItem>>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! cache_get was used.")
    }

    async fn cache_keys(&self, _prefix: String) -> Result<Vec<IdRow<CacheItem>>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! cache_keys was used.")
    }

    async fn cache_incr(&self, _: String) -> Result<IdRow<CacheItem>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! cache_incr was used.")
    }

    async fn queue_all(&self, _limit: Option<usize>) -> Result<Vec<QueueAllItem>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_all was used.")
    }

    async fn queue_results_all(
        &self,
        _limit: Option<usize>,
    ) -> Result<Vec<IdRow<QueueResult>>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_results_all was used.")
    }

    async fn queue_results_multi_delete(&self, _ids: Vec<u64>) -> Result<(), CubeError> {

View on GitHub (pinned to 7d981676b3)

Solutions

Call cache_keys on the master node to list cluster cache entries.
Update monitoring/admin tooling to only query the master for cache listings.
Re-check deployment topology so inspection tools target the head node.

Example fix

// before
let keys = worker_cache_store.cache_keys(prefix).await?; // panics on worker
// after
let keys = master_cache_store.cache_keys(prefix).await?;

Defensive patterns

Strategy: validation

Validate before calling

if node_role() == Role::Worker {
    return Err(CubeError::internal("cache_keys is master-only"));
}
let keys = cache_store.cache_keys(prefix).await?;

Type guard

fn is_master_cache(store: &dyn CacheStore) -> bool {
    std::any::Any::type_id(store) != std::any::TypeId::of::<ClusterCacheStoreClient>()
}

Try / catch

if is_worker_cache_stub(cache_store) {
    return Err(CubeError::internal("list cache keys via the master node"));
}
let keys = cache_store.cache_keys(prefix).await?;

Prevention

Enumerate cache entries only on the master node
Configure dashboards/monitoring to query the head node
Do not broadcast cache listing calls to all nodes
Record node roles in service discovery to ease routing

When it happens

Trigger: Calling cache_keys(prefix) on a worker node's ClusterCacheStoreClient — e.g. admin UI enumerating cache entries and iterating over all cluster nodes.

Common situations: Cache inspection dashboards hitting worker APIs; monitoring scripts broadcasting key-list calls cluster-wide.

Related errors

AI-assisted analysis of cube-js/cube@7d981676b3 (2026-09-02). Data as JSON: /api/errors/f3b6096e31a2715a. Report an issue: GitHub.