CacheStore cannot be used on the worker node! queue_retrieve

Error message

CacheStore cannot be used on the worker node! queue_retrieve_by_path was used.

What it means

CubeStore's CacheStore (the cache-only RocksDB store used on worker nodes) deliberately panics when any queue-related method is invoked. Queue operations (orchestration queue retrieve/ack/result) are only supported on the full cache store used by the master node; calling them on a worker node indicates a routing/deployment bug where queue work was sent to a node that cannot process it. The panic aborts the calling task with this message.

Source

Thrown at rust/cubestore/cubestore/src/cachestore/cache_rocksstore.rs:2048

    async fn queue_get(&self, _key: QueueKey) -> Result<Option<QueueGetResponse>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_get was used.")
    }

    async fn queue_cancel(&self, _key: QueueKey) -> Result<Option<QueueCancelResponse>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_cancel was used.")
    }

    async fn queue_heartbeat(&self, _key: QueueKey) -> Result<(), CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_heartbeat_by_id was used.")
    }

    async fn queue_retrieve_by_path(
        &self,
        _path: String,
        _allow_concurrency: u32,
        _caller_process_id: Option<String>,
    ) -> Result<QueueRetrieveResponse, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_retrieve_by_path was used.")
    }

    async fn queue_ack(&self, _key: QueueKey, _result: Option<String>) -> Result<bool, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_ack was used.")
    }

    async fn queue_result(
        &self,
        _key: QueueKey,
        _external_id: Option<String>,
    ) -> Result<Option<QueueResultResponse>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_result was used.")
    }

    async fn queue_result_blocking(
        &self,
        _key: QueueKey,
        _timeout: u64,

View on GitHub (pinned to 7d981676b3)

Solutions

Verify the queue consumer is connected to the master CubeStore node, not a worker node
Check CubeStore startup flags/cluster config so worker nodes never execute queue retrieval; route queue ops via the master
If this panic is hit in tests, construct the store with the full (non-worker) CacheStore implementation instead of the cache-only one

Example fix

// before: worker node executing queue retrieval locally
let resp = cache_store.queue_retrieve_by_path(path, 1, caller_id).await?;
// after: route queue retrieval to the master node / use the master-backed store
let resp = master_cache_store_client.queue_retrieve_by_path(path, 1, caller_id).await?;

Defensive patterns

Strategy: validation

Validate before calling

if node_role != NodeRole::Master { return Err(CubeError::internal("queue ops require master node")); }
// only then: store.queue_retrieve_by_path(path, 1, caller_id).await?

Type guard

fn is_master_store(store: &dyn CacheStore) -> bool { !std::any::TypeId::of::<CacheRocksStore>().eq(&store.as_any().type_id()) || node_role == NodeRole::Master }

Try / catch

// panics, not Result errors: guard before calling; optionally catch_unwind
let resp = std::panic::catch_unwind(AssertUnwindSafe(|| master_client.queue_retrieve_by_path(...))).map_err(|_| CubeError::internal("queue_retrieve_by_path called on worker node"))?;

Prevention

Run queue consumers only on the master CubeStore node
Assert node role at startup before initializing queue loops
Route all queue ops through a remote client to the master
Tag worker deployments so orchestrators cannot schedule queue work on them

When it happens

Trigger: Calling the CacheStore trait's queue_retrieve_by_path method on a CubeStore worker node instance, where the method is implemented as an unconditional panic! stub (cache_rocksstore.rs:2048).

Common situations: Running CubeStore in worker mode but having the queue/processing logic (e.g. query orchestrator queue retrieval) target the local store instead of the master node; misconfigured cluster topology where a worker was promoted or addressed as master.

Related errors

AI-assisted analysis of cube-js/cube@7d981676b3 (2026-09-02). Data as JSON: /api/errors/f319ab85b90e13f4. Report an issue: GitHub.