{"record":{"id":"c05681cbfdf356e0","repo":"cube-js/cube","slug":"cachestore-cannot-be-used-on-the-worker-node-queu-c05681","errorCode":null,"errorMessage":"CacheStore cannot be used on the worker node! queue_to_cancel was used.","messagePattern":"CacheStore cannot be used on the worker node! queue_to_cancel was used\\.","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"rust/cubestore/cubestore/src/cachestore/cache_rocksstore.rs","lineNumber":2016,"sourceCode":"\n    async fn queue_add_and_retrieve(\n        &self,\n        _payload: QueueAddAndRetrievePayload,\n    ) -> Result<QueueAddAndRetrieveResponse, CubeError> {\n        panic!(\"CacheStore cannot be used on the worker node! queue_add_and_retrieve was used.\")\n    }\n\n    async fn queue_clear(&self) -> Result<(), CubeError> {\n        panic!(\"CacheStore cannot be used on the worker node! queue_clear was used.\")\n    }\n\n    async fn queue_to_cancel(\n        &self,\n        _prefix: String,\n        _orphaned_timeout: Option<u32>,\n        _heartbeat_timeout: Option<u32>,\n    ) -> Result<Vec<IdRow<QueueItem>>, CubeError> {\n        panic!(\"CacheStore cannot be used on the worker node! queue_to_cancel was used.\")\n    }\n\n    async fn queue_list(\n        &self,\n        _prefix: String,\n        _status_filter: Option<QueueItemStatus>,\n        _priority_sort: bool,\n        _with_payload: bool,\n        _caller_process_id: Option<String>,\n    ) -> Result<Vec<QueueListItem>, CubeError> {\n        panic!(\"CacheStore cannot be used on the worker node! queue_list was used.\")\n    }\n\n    async fn queue_get(&self, _key: QueueKey) -> Result<Option<QueueGetResponse>, CubeError> {\n        panic!(\"CacheStore cannot be used on the worker node! queue_get was used.\")\n    }\n\n    async fn queue_cancel(&self, _key: QueueKey) -> Result<Option<QueueCancelResponse>, CubeError> {","sourceCodeStart":1998,"sourceCodeEnd":2034,"githubUrl":"https://github.com/cube-js/cube/blob/7d981676b36392fec34088b9afab6bdcad40207c/rust/cubestore/cubestore/src/cachestore/cache_rocksstore.rs#L1998-L2034","documentation":"queue_to_cancel scans for queue items that should be cancelled (by prefix and timeouts) as part of queue housekeeping on the metastore. CacheStore on a worker panics because workers never own queue items.","triggerScenarios":"A worker node executing the queue housekeeping/cancellation scan (queue_to_cancel with prefix, orphaned_timeout, heartbeat_timeout).","commonSituations":"Cancellation/orphan-reaping loops running on every node instead of only the metastore; replicated scheduler processes each believing they are the leader.","solutions":["Run queue housekeeping (including cancellation scans) only on the metastore node.","Add leader election / role gating so worker processes skip queue maintenance tasks.","Confirm the process performing this call has metastore storage attached.","Check deployment manifests that a queue-worker sidecar isn't attached to worker pods."],"exampleFix":"// before: every node runs housekeeping\nfor node in allNodes { node.queueToCancel(prefix, t).await; }\n// after\nif isMetastore { metastore.queueToCancel(prefix, t).await; }","handlingStrategy":"validation","validationCode":"if !is_metastore(node) {\n    // workers must skip queue housekeeping entirely\n    return Ok(());\n}\nnode.queueToCancel(prefix, orphaned_timeout, heartbeat_timeout).await?;","typeGuard":"fn is_metastore(node: &CubeStoreNode) -> bool {\n    matches!(node.role(), NodeRole::Metastore)\n}","tryCatchPattern":null,"preventionTips":["Gate housekeeping loops behind leader/metastore role checks.","Avoid running identical schedulers on every node.","Inspect deployment manifests for stray queue-worker sidecars."],"tags":["cubestore","rust","topology","queue"],"backgroundTag":"wrong-node-role","analyzedSha":"7d981676b36392fec34088b9afab6bdcad40207c","analyzedAt":"2026-09-02T03:45:10.400Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}