xai-org/grok-build · error · std::io::Error

workflow persistence actor dropped acknowledgement

Error message

workflow persistence actor dropped acknowledgement

What it means

After successfully sending PersistenceMsg::WorkflowRunStateAndAck, persist_ack awaits the oneshot respond_to acknowledgement. If the persistence actor is dropped (or its task aborts) before replying, the oneshot future resolves with a RecvError and this BrokenPipe io::Error is returned.

Source

Thrown at crates/codegen/xai-grok-shell/src/session/workflow/store.rs:227

            io::Error::new(
                io::ErrorKind::NotFound,
                "workflow state has no registered resume source",
            )
        })?;
        let (respond_to, response) = oneshot::channel();
        self.persistence_tx
            .send(PersistenceMsg::WorkflowRunStateAndAck {
                manifest,
                respond_to,
            })
            .map_err(|_| {
                io::Error::new(
                    io::ErrorKind::BrokenPipe,
                    "workflow persistence channel closed",
                )
            })?;
        response.await.map_err(|_| {
            io::Error::new(
                io::ErrorKind::BrokenPipe,
                "workflow persistence actor dropped acknowledgement",
            )
        })?
    }

    pub(crate) fn remove(&self, run_id: &str) {
        self.sources.lock().remove(run_id);
        if let Some(run_dir) = self.run_dir(run_id) {
            if let Err(error) = atomic_write_replace(&run_dir.join("cleared"), b"") {
                tracing::warn!(run_id, %error, "failed to tombstone cleared workflow run");
            }
            if let Err(error) = std::fs::remove_file(run_dir.join("state.json"))
                && error.kind() != io::ErrorKind::NotFound
            {
                tracing::warn!(run_id, %error, "failed to remove workflow manifest during clear");
            }
        }

View on GitHub (pinned to bc7f02eddd)

Solutions

Inspect persistence actor code for panics or paths where the oneshot responder is dropped without send
Keep the actor task alive across the request lifetime (don't abort it mid-flight)
Ensure the actor's message handler always replies to respond_to (including on internal error)
Add catch_unwind/JoinHandle monitoring around the actor so shutdown is orderly

Example fix

// before
// actor: if let Ok(...) = tx.send(...) { respond_to.send(()) }; // dropped on early return
// after
let result = do_persist(&manifest);
let _ = respond_to.send(result); // always ack

Defensive patterns

Strategy: try-catch

Validate before calling

// no pre-call validation is possible; ensure actor responds on all paths
// invariant check in actor handler:
// every arm of PersistenceMsg handling must call respond_to.send(...) exactly once

Type guard

fn is_actor_dropped(e: &std::io::Error) -> bool {
    e.kind() == std::io::ErrorKind::BrokenPipe
        && e.to_string().contains("dropped acknowledgement")
}

Try / catch

let ack = store.persist_ack(manifest, respond_to).await.map_err(|e| {
    if e.to_string().contains("dropped acknowledgement") {
        PersistError::ActorCrashed(e)
    } else {
        PersistError::Io(e)
    }
})?;

Prevention

Make the actor handler reply to every message, including error paths
Wrap actor internals to convert panics into an error reply, not a drop
Avoid aborting the actor task while requests are in flight
Add tests that kill the actor mid-persist and assert the error kind

When it happens

Trigger: The actor receives the message but exits before calling respond_to: task aborted at shutdown, panic inside the actor's handling of WorkflowRunStateAndAck, or the actor deliberately drops the message without replying.

Common situations: Actor panic while writing workflow state to disk; process shutdown racing an in-flight ack; actor loop that only replies on some code paths.

Related errors

AI-assisted analysis of xai-org/grok-build@bc7f02eddd (2026-08-31). Data as JSON: /api/errors/ef1a8135dec0742f. Report an issue: GitHub.