CacheStore cannot be used on the worker node! cache_incr was

Error message

CacheStore cannot be used on the worker node! cache_incr was used.

What it means

ClusterCacheStoreClient (worker-node CacheStore) panics on cache_incr; atomic counter increments on cache items must be performed by the master that owns the underlying store. Workers panic to prevent split-brain counter updates.

Source

Thrown at rust/cubestore/cubestore/src/cachestore/cache_rocksstore.rs:1977

    async fn truncate(&self) -> Result<(), CubeError> {
        panic!("CacheStore cannot be used on the worker node! truncate was used.")
    }

    async fn cache_delete(&self, _key: String) -> Result<(), CubeError> {
        panic!("CacheStore cannot be used on the worker node! cache_delete was used.")
    }

    async fn cache_get(&self, _key: String) -> Result<Option<IdRow<CacheItem>>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! cache_get was used.")
    }

    async fn cache_keys(&self, _prefix: String) -> Result<Vec<IdRow<CacheItem>>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! cache_keys was used.")
    }

    async fn cache_incr(&self, _: String) -> Result<IdRow<CacheItem>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! cache_incr was used.")
    }

    async fn queue_all(&self, _limit: Option<usize>) -> Result<Vec<QueueAllItem>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_all was used.")
    }

    async fn queue_results_all(
        &self,
        _limit: Option<usize>,
    ) -> Result<Vec<IdRow<QueueResult>>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_results_all was used.")
    }

    async fn queue_results_multi_delete(&self, _ids: Vec<u64>) -> Result<(), CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_results_multi_delete was used.")
    }

    async fn queue_add(&self, _payload: QueueAddPayload) -> Result<QueueAddResponse, CubeError> {

View on GitHub (pinned to 7d981676b3)

Solutions

Route cache_incr to the master node or the cluster cache client so increments are atomic at the owner.
Redesign the counter path to use the cluster API instead of per-node CacheStore calls.
Verify which node the incrementing component runs on and its configured role.

Example fix

// before
worker_cache_store.cache_incr(key).await?; // panics on worker
// after
cluster.master_cache_store().cache_incr(key).await?;

Defensive patterns

Strategy: validation

Validate before calling

if node_role() == Role::Worker {
    return Err(CubeError::internal("cache_incr is master-only"));
}
cache_store.cache_incr(key).await?;

Type guard

fn is_master_cache(store: &dyn CacheStore) -> bool {
    std::any::Any::type_id(store) != std::any::TypeId::of::<ClusterCacheStoreClient>()
}

Try / catch

if is_worker_cache_stub(cache_store) {
    return Err(CubeError::internal("cache_incr must be executed on the master node"));
}
cache_store.cache_incr(key).await?;

Prevention

Perform atomic counter increments on the master store only
Design counters to use the cluster API from any node
Confirm the incrementing component's node role at startup
Never share worker-local CacheStore handles for mutations

When it happens

Trigger: Calling cache_incr(key) on a worker node's ClusterCacheStoreClient — e.g. usage counters or rate-limit bookkeeping routed to a worker.

Common situations: Counter/semaphore logic accidentally using the worker-local store; distributed rate limiting implemented against the wrong node.

Related errors

AI-assisted analysis of cube-js/cube@7d981676b3 (2026-09-02). Data as JSON: /api/errors/8bb71756d7946c5b. Report an issue: GitHub.