CacheStore cannot be used on the worker node! queue_all was

Error message

CacheStore cannot be used on the worker node! queue_all was used.

What it means

ClusterCacheStoreClient (worker-node CacheStore) panics on queue_all; scanning the ingestion/query queue is master-only because the queue lives in the master's metastore-backed cache. The worker implementation intentionally unimplemented to catch misrouted queue operations.

Source

Thrown at rust/cubestore/cubestore/src/cachestore/cache_rocksstore.rs:1981

    async fn cache_delete(&self, _key: String) -> Result<(), CubeError> {
        panic!("CacheStore cannot be used on the worker node! cache_delete was used.")
    }

    async fn cache_get(&self, _key: String) -> Result<Option<IdRow<CacheItem>>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! cache_get was used.")
    }

    async fn cache_keys(&self, _prefix: String) -> Result<Vec<IdRow<CacheItem>>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! cache_keys was used.")
    }

    async fn cache_incr(&self, _: String) -> Result<IdRow<CacheItem>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! cache_incr was used.")
    }

    async fn queue_all(&self, _limit: Option<usize>) -> Result<Vec<QueueAllItem>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_all was used.")
    }

    async fn queue_results_all(
        &self,
        _limit: Option<usize>,
    ) -> Result<Vec<IdRow<QueueResult>>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_results_all was used.")
    }

    async fn queue_results_multi_delete(&self, _ids: Vec<u64>) -> Result<(), CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_results_multi_delete was used.")
    }

    async fn queue_add(&self, _payload: QueueAddPayload) -> Result<QueueAddResponse, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_add was used.")
    }

    async fn queue_add_and_retrieve(

View on GitHub (pinned to 7d981676b3)

Solutions

Fetch queue items via the master node or the cluster queue client (the intended path for workers).
Ensure worker nodes receive work through the cluster protocol rather than calling CacheStore::queue_all directly.
Audit startup code so the correct CacheStore/queue implementation is injected per node role.

Example fix

// before
let items = worker_cache_store.queue_all(limit).await?; // panics on worker
// after
let items = cluster.queue_client().queue_all(limit).await?; // cluster path

Defensive patterns

Strategy: validation

Validate before calling

if node_role() == Role::Worker {
    return Err(CubeError::internal("queue_all is master-only; use the cluster queue client"));
}
let items = cache_store.queue_all(limit).await?;

Type guard

fn is_master_cache(store: &dyn CacheStore) -> bool {
    std::any::Any::type_id(store) != std::any::TypeId::of::<ClusterCacheStoreClient>()
}

Try / catch

if is_worker_cache_stub(cache_store) {
    return Err(CubeError::internal("fetch queue items via the cluster queue client"));
}
let items = cache_store.queue_all(limit).await?;

Prevention

Workers should receive queue work via the cluster protocol, not CacheStore::queue_all
Inject the queue client appropriate to the node role at startup
Audit background loops for direct CacheStore queue calls
Keep queue-drain logic on the master or use the distributed path

When it happens

Trigger: Calling queue_all(limit) on a worker node's ClusterCacheStoreClient — e.g. queue-drain or job-fetch logic running on a worker instead of the master.

Common situations: Background worker loops instantiated on worker nodes that incorrectly use the local CacheStore for queue fetching; scheduler misconfiguration.

Related errors

AI-assisted analysis of cube-js/cube@7d981676b3 (2026-09-02). Data as JSON: /api/errors/3fcf11fb7a8086fa. Report an issue: GitHub.