xai-org/grok-build · error · std::io::Error

workflow persistence channel closed

Error message

workflow persistence channel closed

What it means

persist sends PersistenceMsg::WorkflowRunState over a bounded mpsc channel to the persistence worker; if the receiver has been dropped (worker thread exited/panicked or store shutdown completed), send fails and the library maps it to io::ErrorKind::BrokenPipe "workflow persistence channel closed". The state update is then not durably persisted.

Source

Thrown at crates/codegen/xai-grok-shell/src/session/workflow/store.rs:200

        };
        let Some(run_dir) = self.run_dir(&state.run_id) else {
            return Ok(());
        };
        let json = serde_json::to_vec_pretty(&manifest).map_err(io::Error::other)?;
        atomic_write_replace(&run_dir.join("state.json"), &json)
    }

    pub(crate) fn persist(&self, state: &WorkflowRunState) -> io::Result<()> {
        let manifest = self.manifest_for(state).ok_or_else(|| {
            io::Error::new(
                io::ErrorKind::NotFound,
                "workflow state has no registered resume source",
            )
        })?;
        self.persistence_tx
            .send(PersistenceMsg::WorkflowRunState(manifest))
            .map_err(|_| {
                io::Error::new(
                    io::ErrorKind::BrokenPipe,
                    "workflow persistence channel closed",
                )
            })
    }

    pub(crate) async fn persist_ack(&self, state: &WorkflowRunState) -> io::Result<()> {
        let manifest = self.manifest_for(state).ok_or_else(|| {
            io::Error::new(
                io::ErrorKind::NotFound,
                "workflow state has no registered resume source",
            )
        })?;
        let (respond_to, response) = oneshot::channel();
        self.persistence_tx
            .send(PersistenceMsg::WorkflowRunStateAndAck {
                manifest,
                respond_to,

View on GitHub (pinned to bc7f02eddd)

Solutions

1. Check store shutdown state and skip persist during teardown (treat BrokenPipe as benign there)
2. Restart/respawn the persistence worker thread if it exited unexpectedly
3. Fall back to persist_now (synchronous write) when the channel send fails
4. Log and surface worker panics so the channel doesn't close silently

Example fix

// before
store.persist(&state)?; // BrokenPipe during shutdown
// after
if let Err(e) = store.persist(&state) {
    if e.kind() == io::ErrorKind::BrokenPipe && !shutting_down {
        store.persist_now(&state)?;
    }
}

// channel health cannot be pre-validated; detect via send result
// ensure store is not shutting down before persisting
if !store.is_running() { return Ok(()); }

fn store_accepts_persist(store: &WorkflowStore) -> bool {
    !store.is_shutting_down()
}

match store.persist(&state) {
    Err(e) if e.kind() == io::ErrorKind::BrokenPipe => {
        // worker gone: fall back to synchronous write or log-and-drop during shutdown
        store.persist_now(&state)?;
    }
    other => other?,
}

When it happens

Trigger: Calling persist after the persistence worker thread has terminated — during/after shutdown, after a panic in the persistence loop, or if the store's worker was never spawned.

Common situations: A shutdown race where a task persists state while the store is being dropped; persistence thread crashed earlier on an unrelated write error; long-running process where the worker exited silently.

Related errors

• blocking write task panicked: {e}
• transport is closed
• {}
• cancelled during ignored-only copy
• cancelled after git worktree add

AI-assisted analysis of xai-org/grok-build@bc7f02eddd (2026-08-31). Data as JSON: /api/errors/8d585dc7e8d46adc. Report an issue: GitHub.