CacheStore cannot be used on the worker node! queue_ack was

Error message

CacheStore cannot be used on the worker node! queue_ack was used.

What it means

Same family as the other CacheStore worker-node stubs: on a worker node the queue_ack implementation is a deliberate panic! because worker nodes hold only a cache store and cannot acknowledge queue messages. Reaching this code means an acknowledgment was attempted against a store that does not own the queue.

Source

Thrown at rust/cubestore/cubestore/src/cachestore/cache_rocksstore.rs:2052

    async fn queue_cancel(&self, _key: QueueKey) -> Result<Option<QueueCancelResponse>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_cancel was used.")
    }

    async fn queue_heartbeat(&self, _key: QueueKey) -> Result<(), CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_heartbeat_by_id was used.")
    }

    async fn queue_retrieve_by_path(
        &self,
        _path: String,
        _allow_concurrency: u32,
        _caller_process_id: Option<String>,
    ) -> Result<QueueRetrieveResponse, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_retrieve_by_path was used.")
    }

    async fn queue_ack(&self, _key: QueueKey, _result: Option<String>) -> Result<bool, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_ack was used.")
    }

    async fn queue_result(
        &self,
        _key: QueueKey,
        _external_id: Option<String>,
    ) -> Result<Option<QueueResultResponse>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_result was used.")
    }

    async fn queue_result_blocking(
        &self,
        _key: QueueKey,
        _timeout: u64,
    ) -> Result<Option<QueueResultResponse>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_result_blocking was used.")
    }

View on GitHub (pinned to 7d981676b3)

Solutions

Send queue_ack to the master node's store (the store that issued the queue item), not the local worker store
Audit cluster configuration so only master-capable nodes implement queue ack; workers should proxy to master
Replace the cache-only CacheStore with the full implementation in any process expected to ack queue messages

Example fix

// before
local_store.queue_ack(key, Some(result)).await?;
// after
master_client.queue_ack(key, Some(result)).await?;

Defensive patterns

Strategy: validation

Validate before calling

if !is_master { return Err(CubeError::internal("queue_ack requires master node")); }
master_client.queue_ack(key, result).await?;

Type guard

fn supports_queue_ops(store: &dyn CacheStore) -> bool { /* true only for the full master store impl */ matches!(node_role, NodeRole::Master) }

Try / catch

// panic-based guard: wrap in catch_unwind if the call site cannot be proven
let ok = std::panic::catch_unwind(AssertUnwindSafe(|| master_client.queue_ack(key, result))).map_err(|_| CubeError::internal("queue_ack called on worker node"))?;

Prevention

Ack queue items on the same node/store that issued them
Keep worker nodes free of queue-ack code paths
Use remote master clients in shared queue logic
Add integration tests covering master/worker routing

When it happens

Trigger: Invoking CacheStore::queue_ack(key, result) on a worker-node store instance (cache_rocksstore.rs:2052) after processing a queue item, when the queue actually lives on the master.

Common situations: Misrouted queue acknowledgement in a CubeStore cluster; code that consumed a queue item via a remote client but acked via the local worker store; test harness wiring the cache-only store into queue logic.

Related errors

AI-assisted analysis of cube-js/cube@7d981676b3 (2026-09-02). Data as JSON: /api/errors/8b98ea195c7b79f3. Report an issue: GitHub.