{"record":{"id":"ef1a8135dec0742f","repo":"xai-org/grok-build","slug":"workflow-persistence-actor-dropped-acknowledgement","errorCode":null,"errorMessage":"workflow persistence actor dropped acknowledgement","messagePattern":"workflow persistence actor dropped acknowledgement","errorType":"exception","errorClass":"std::io::Error","httpStatus":null,"severity":"error","filePath":"crates/codegen/xai-grok-shell/src/session/workflow/store.rs","lineNumber":227,"sourceCode":"            io::Error::new(\n                io::ErrorKind::NotFound,\n                \"workflow state has no registered resume source\",\n            )\n        })?;\n        let (respond_to, response) = oneshot::channel();\n        self.persistence_tx\n            .send(PersistenceMsg::WorkflowRunStateAndAck {\n                manifest,\n                respond_to,\n            })\n            .map_err(|_| {\n                io::Error::new(\n                    io::ErrorKind::BrokenPipe,\n                    \"workflow persistence channel closed\",\n                )\n            })?;\n        response.await.map_err(|_| {\n            io::Error::new(\n                io::ErrorKind::BrokenPipe,\n                \"workflow persistence actor dropped acknowledgement\",\n            )\n        })?\n    }\n\n    pub(crate) fn remove(&self, run_id: &str) {\n        self.sources.lock().remove(run_id);\n        if let Some(run_dir) = self.run_dir(run_id) {\n            if let Err(error) = atomic_write_replace(&run_dir.join(\"cleared\"), b\"\") {\n                tracing::warn!(run_id, %error, \"failed to tombstone cleared workflow run\");\n            }\n            if let Err(error) = std::fs::remove_file(run_dir.join(\"state.json\"))\n                && error.kind() != io::ErrorKind::NotFound\n            {\n                tracing::warn!(run_id, %error, \"failed to remove workflow manifest during clear\");\n            }\n        }","sourceCodeStart":209,"sourceCodeEnd":245,"githubUrl":"https://github.com/xai-org/grok-build/blob/bc7f02eddd3d84085849dc19ed216f11c23b0571/crates/codegen/xai-grok-shell/src/session/workflow/store.rs#L209-L245","documentation":"After successfully sending PersistenceMsg::WorkflowRunStateAndAck, persist_ack awaits the oneshot respond_to acknowledgement. If the persistence actor is dropped (or its task aborts) before replying, the oneshot future resolves with a RecvError and this BrokenPipe io::Error is returned.","triggerScenarios":"The actor receives the message but exits before calling respond_to: task aborted at shutdown, panic inside the actor's handling of WorkflowRunStateAndAck, or the actor deliberately drops the message without replying.","commonSituations":"Actor panic while writing workflow state to disk; process shutdown racing an in-flight ack; actor loop that only replies on some code paths.","solutions":["Inspect persistence actor code for panics or paths where the oneshot responder is dropped without send","Keep the actor task alive across the request lifetime (don't abort it mid-flight)","Ensure the actor's message handler always replies to respond_to (including on internal error)","Add catch_unwind/JoinHandle monitoring around the actor so shutdown is orderly"],"exampleFix":"// before\n// actor: if let Ok(...) = tx.send(...) { respond_to.send(()) }; // dropped on early return\n// after\nlet result = do_persist(&manifest);\nlet _ = respond_to.send(result); // always ack\n","handlingStrategy":"try-catch","validationCode":"// no pre-call validation is possible; ensure actor responds on all paths\n// invariant check in actor handler:\n// every arm of PersistenceMsg handling must call respond_to.send(...) exactly once","typeGuard":"fn is_actor_dropped(e: &std::io::Error) -> bool {\n    e.kind() == std::io::ErrorKind::BrokenPipe\n        && e.to_string().contains(\"dropped acknowledgement\")\n}","tryCatchPattern":"let ack = store.persist_ack(manifest, respond_to).await.map_err(|e| {\n    if e.to_string().contains(\"dropped acknowledgement\") {\n        PersistError::ActorCrashed(e)\n    } else {\n        PersistError::Io(e)\n    }\n})?;","preventionTips":["Make the actor handler reply to every message, including error paths","Wrap actor internals to convert panics into an error reply, not a drop","Avoid aborting the actor task while requests are in flight","Add tests that kill the actor mid-persist and assert the error kind"],"tags":["io","broken-pipe","oneshot","ack"],"backgroundTag":"channel-closed","analyzedSha":"bc7f02eddd3d84085849dc19ed216f11c23b0571","analyzedAt":"2026-08-31T04:59:42.031Z","schemaVersion":2},"datasetVersion":"2026-08-31T09:17:48.483Z"}