CacheStore cannot be used on the worker node! queue_to_cance

Error message

CacheStore cannot be used on the worker node! queue_to_cancel was used.

What it means

queue_to_cancel scans for queue items that should be cancelled (by prefix and timeouts) as part of queue housekeeping on the metastore. CacheStore on a worker panics because workers never own queue items.

Source

Thrown at rust/cubestore/cubestore/src/cachestore/cache_rocksstore.rs:2016

    async fn queue_add_and_retrieve(
        &self,
        _payload: QueueAddAndRetrievePayload,
    ) -> Result<QueueAddAndRetrieveResponse, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_add_and_retrieve was used.")
    }

    async fn queue_clear(&self) -> Result<(), CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_clear was used.")
    }

    async fn queue_to_cancel(
        &self,
        _prefix: String,
        _orphaned_timeout: Option<u32>,
        _heartbeat_timeout: Option<u32>,
    ) -> Result<Vec<IdRow<QueueItem>>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_to_cancel was used.")
    }

    async fn queue_list(
        &self,
        _prefix: String,
        _status_filter: Option<QueueItemStatus>,
        _priority_sort: bool,
        _with_payload: bool,
        _caller_process_id: Option<String>,
    ) -> Result<Vec<QueueListItem>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_list was used.")
    }

    async fn queue_get(&self, _key: QueueKey) -> Result<Option<QueueGetResponse>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_get was used.")
    }

    async fn queue_cancel(&self, _key: QueueKey) -> Result<Option<QueueCancelResponse>, CubeError> {

View on GitHub (pinned to 7d981676b3)

Solutions

Run queue housekeeping (including cancellation scans) only on the metastore node.
Add leader election / role gating so worker processes skip queue maintenance tasks.
Confirm the process performing this call has metastore storage attached.
Check deployment manifests that a queue-worker sidecar isn't attached to worker pods.

Example fix

// before: every node runs housekeeping
for node in allNodes { node.queueToCancel(prefix, t).await; }
// after
if isMetastore { metastore.queueToCancel(prefix, t).await; }

Defensive patterns

Strategy: validation

Validate before calling

if !is_metastore(node) {
    // workers must skip queue housekeeping entirely
    return Ok(());
}
node.queueToCancel(prefix, orphaned_timeout, heartbeat_timeout).await?;

Type guard

fn is_metastore(node: &CubeStoreNode) -> bool {
    matches!(node.role(), NodeRole::Metastore)
}

Prevention

Gate housekeeping loops behind leader/metastore role checks.
Avoid running identical schedulers on every node.
Inspect deployment manifests for stray queue-worker sidecars.

When it happens

Trigger: A worker node executing the queue housekeeping/cancellation scan (queue_to_cancel with prefix, orphaned_timeout, heartbeat_timeout).

Common situations: Cancellation/orphan-reaping loops running on every node instead of only the metastore; replicated scheduler processes each believing they are the leader.

Related errors

AI-assisted analysis of cube-js/cube@7d981676b3 (2026-09-02). Data as JSON: /api/errors/c05681cbfdf356e0. Report an issue: GitHub.