CacheStore cannot be used on the worker node! queue_heartbea

Error message

CacheStore cannot be used on the worker node! queue_heartbeat_by_id was used.

What it means

queue_heartbeat renews a queue item's lease so it isn't considered orphaned; it must run against the metastore's queue state. CacheStore on a worker panics (note the message says queue_heartbeat_by_id, a naming inconsistency in the panic string).

Source

Thrown at rust/cubestore/cubestore/src/cachestore/cache_rocksstore.rs:2039

        _prefix: String,
        _status_filter: Option<QueueItemStatus>,
        _priority_sort: bool,
        _with_payload: bool,
        _caller_process_id: Option<String>,
    ) -> Result<Vec<QueueListItem>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_list was used.")
    }

    async fn queue_get(&self, _key: QueueKey) -> Result<Option<QueueGetResponse>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_get was used.")
    }

    async fn queue_cancel(&self, _key: QueueKey) -> Result<Option<QueueCancelResponse>, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_cancel was used.")
    }

    async fn queue_heartbeat(&self, _key: QueueKey) -> Result<(), CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_heartbeat_by_id was used.")
    }

    async fn queue_retrieve_by_path(
        &self,
        _path: String,
        _allow_concurrency: u32,
        _caller_process_id: Option<String>,
    ) -> Result<QueueRetrieveResponse, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_retrieve_by_path was used.")
    }

    async fn queue_ack(&self, _key: QueueKey, _result: Option<String>) -> Result<bool, CubeError> {
        panic!("CacheStore cannot be used on the worker node! queue_ack was used.")
    }

    async fn queue_result(
        &self,
        _key: QueueKey,

View on GitHub (pinned to 7d981676b3)

Solutions

Point heartbeats at the metastore node that owns the queue item.
Update heartbeat target after any metastore failover/migration.
Ensure worker processes only fetch/process items and never renew leases via a worker store.
If repeated, check that the item still exists — heartbeat calls for cancelled items may indicate stale keys.

Example fix

// before: heartbeat routed to worker store
heartbeat_target = worker_addr;
// after
heartbeat_target = metastore_addr;

Defensive patterns

Strategy: validation

Validate before calling

if !is_metastore(node) {
    return Err(CubeError::internal("queue_heartbeat requires the metastore node"));
}
node.queueHeartbeat(key).await?;

Type guard

fn is_metastore(node: &CubeStoreNode) -> bool {
    matches!(node.role(), NodeRole::Metastore)
}

Prevention

Update heartbeat targets after metastore failover.
Only workers' processing logic (backed by metastore connection) should renew leases.
Alert on heartbeat failures — silent misrouting causes spurious orphan-cancellation of running builds.

When it happens

Trigger: A worker processing loop or client calling queue_heartbeat (QueueKey) against CacheStore — e.g. heartbeat logic misrouted from a build worker to the wrong node.

Common situations: Heartbeat timers configured with the address of a cache/worker node; after topology changes the heartbeat target wasn't updated to the metastore; long-running pre-aggregation builds whose keep-alive hits the wrong node, causing spurious orphaning.

Related errors

AI-assisted analysis of cube-js/cube@7d981676b3 (2026-09-02). Data as JSON: /api/errors/1bd218e960359a5b. Report an issue: GitHub.