CacheStore cannot be used on the worker node! queue_results_

Error message

CacheStore cannot be used on the worker node! queue_results_multi_delete was used.

What it means

CacheStore on a CubeStore worker node cannot perform queue maintenance; queue_results_multi_delete (bulk deletion of finished queue results) is implemented only by the real metastore store and panics on the worker. This is a deliberate guard so queue state is never mutated outside the metastore.

Source

Thrown at rust/cubestore/cubestore/src/cachestore/cache_rocksstore.rs:1992

    }

    async fn cache_incr(&self, _: String) -> Result<IdRow<CacheItem>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! cache_incr was used.")
    }

    async fn queue_all(&self, _limit: Option<usize>) -> Result<Vec<QueueAllItem>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_all was used.")
    }

    async fn queue_results_all(
        &self,
        _limit: Option<usize>,
    ) -> Result<Vec<IdRow<QueueResult>>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_results_all was used.")
    }

    async fn queue_results_multi_delete(&self, _ids: Vec<u64>) -> Result<(), CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_results_multi_delete was used.")
    }

    async fn queue_add(&self, _payload: QueueAddPayload) -> Result<QueueAddResponse, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_add was used.")
    }

    async fn queue_add_and_retrieve(
        &self,
        _payload: QueueAddAndRetrievePayload,
    ) -> Result<QueueAddAndRetrieveResponse, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_add_and_retrieve was used.")
    }

    async fn queue_clear(&self) -> Result<(), CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_clear was used.")
    }

    async fn queue_to_cancel(

View on GitHub (pinned to 7d981676b3)

Solutions

Send queue result deletion to the metastore node only.
Exclude worker nodes from any load balancer / client pool used for queue management APIs.
Run cleanup logic on or via the metastore leader process.
Verify node role at startup and skip queue cleanup on workers.

Example fix

// before: cleanup job hits every node incl. workers
for node in nodes { node.queueResultsMultiDelete(ids); }
// after: only metastore handles queue writes
metastoreNode.queueResultsMultiDelete(ids);

Defensive patterns

Strategy: validation

Validate before calling

if !is_metastore(node) {
    return Err(CubeError::internal("queue_results_multi_delete requires the metastore node"));
}
node.queueResultsMultiDelete(ids).await?;

Type guard

fn is_metastore(node: &CubeStoreNode) -> bool {
    matches!(node.role(), NodeRole::Metastore)
}

Prevention

Run result cleanup jobs on the metastore node only.
Exclude workers from pools used for mutating queue APIs.
Audit cron/systemd units so cleanup targets the metastore address.

When it happens

Trigger: Calling queue_results_multi_delete (bulk cleanup of queue results by ids) against a node whose store is CacheStore, i.e. a worker node.

Common situations: Cleanup/retention jobs or queue-trimming code that iterates cluster nodes and hits workers; load balancers distributing metastore API requests to worker replicas.

Related errors

AI-assisted analysis of cube-js/cube@7d981676b3 (2026-09-02). Data as JSON: /api/errors/a8f57ecbcdb1f5c4. Report an issue: GitHub.