{"record":{"id":"8d585dc7e8d46adc","repo":"xai-org/grok-build","slug":"workflow-persistence-channel-closed","errorCode":null,"errorMessage":"workflow persistence channel closed","messagePattern":"workflow persistence channel closed","errorType":"exception","errorClass":"std::io::Error","httpStatus":null,"severity":"error","filePath":"crates/codegen/xai-grok-shell/src/session/workflow/store.rs","lineNumber":200,"sourceCode":"        };\n        let Some(run_dir) = self.run_dir(&state.run_id) else {\n            return Ok(());\n        };\n        let json = serde_json::to_vec_pretty(&manifest).map_err(io::Error::other)?;\n        atomic_write_replace(&run_dir.join(\"state.json\"), &json)\n    }\n\n    pub(crate) fn persist(&self, state: &WorkflowRunState) -> io::Result<()> {\n        let manifest = self.manifest_for(state).ok_or_else(|| {\n            io::Error::new(\n                io::ErrorKind::NotFound,\n                \"workflow state has no registered resume source\",\n            )\n        })?;\n        self.persistence_tx\n            .send(PersistenceMsg::WorkflowRunState(manifest))\n            .map_err(|_| {\n                io::Error::new(\n                    io::ErrorKind::BrokenPipe,\n                    \"workflow persistence channel closed\",\n                )\n            })\n    }\n\n    pub(crate) async fn persist_ack(&self, state: &WorkflowRunState) -> io::Result<()> {\n        let manifest = self.manifest_for(state).ok_or_else(|| {\n            io::Error::new(\n                io::ErrorKind::NotFound,\n                \"workflow state has no registered resume source\",\n            )\n        })?;\n        let (respond_to, response) = oneshot::channel();\n        self.persistence_tx\n            .send(PersistenceMsg::WorkflowRunStateAndAck {\n                manifest,\n                respond_to,","sourceCodeStart":182,"sourceCodeEnd":218,"githubUrl":"https://github.com/xai-org/grok-build/blob/bc7f02eddd3d84085849dc19ed216f11c23b0571/crates/codegen/xai-grok-shell/src/session/workflow/store.rs#L182-L218","documentation":"persist sends PersistenceMsg::WorkflowRunState over a bounded mpsc channel to the persistence worker; if the receiver has been dropped (worker thread exited/panicked or store shutdown completed), send fails and the library maps it to io::ErrorKind::BrokenPipe \"workflow persistence channel closed\". The state update is then not durably persisted.","triggerScenarios":"Calling persist after the persistence worker thread has terminated — during/after shutdown, after a panic in the persistence loop, or if the store's worker was never spawned.","commonSituations":"A shutdown race where a task persists state while the store is being dropped; persistence thread crashed earlier on an unrelated write error; long-running process where the worker exited silently.","solutions":["Check store shutdown state and skip persist during teardown (treat BrokenPipe as benign there)","Restart/respawn the persistence worker thread if it exited unexpectedly","Fall back to persist_now (synchronous write) when the channel send fails","Log and surface worker panics so the channel doesn't close silently"],"exampleFix":"// before\nstore.persist(&state)?; // BrokenPipe during shutdown\n// after\nif let Err(e) = store.persist(&state) {\n    if e.kind() == io::ErrorKind::BrokenPipe && !shutting_down {\n        store.persist_now(&state)?;\n    }\n}","handlingStrategy":"fallback","validationCode":"// channel health cannot be pre-validated; detect via send result\n// ensure store is not shutting down before persisting\nif !store.is_running() { return Ok(()); }","typeGuard":"fn store_accepts_persist(store: &WorkflowStore) -> bool {\n    !store.is_shutting_down()\n}","tryCatchPattern":"match store.persist(&state) {\n    Err(e) if e.kind() == io::ErrorKind::BrokenPipe => {\n        // worker gone: fall back to synchronous write or log-and-drop during shutdown\n        store.persist_now(&state)?;\n    }\n    other => other?,\n}","preventionTips":["Order shutdown: stop producers before dropping the persistence worker","Catch panics in the persistence loop and respawn the worker","Fall back to persist_now when the channel is closed and durability matters","Distinguish shutdown-time BrokenPipe (benign) from mid-run BrokenPipe (bug)"],"tags":["io","channel","shutdown-race","async"],"backgroundTag":"channel-closed","analyzedSha":"bc7f02eddd3d84085849dc19ed216f11c23b0571","analyzedAt":"2026-08-31T04:59:42.031Z","schemaVersion":2},"datasetVersion":"2026-08-31T09:17:48.483Z"}