xai-org/grok-build · error · io::Error

resources persistence writer dropped acknowledgement

Error message

resources persistence writer dropped acknowledgement

What it means

await_save_and_flush awaits the oneshot receiver returned by enqueue_save_and_flush; if the sender side is dropped without replying, it maps the RecvError to BrokenPipe with "resources persistence writer dropped acknowledgement". Unlike error 706 (send failed), here the command was delivered but the writer never sent back a result, so durability is unknown.

Source

Thrown at crates/codegen/xai-grok-tools/src/persistence.rs:180

            .send(ResourcesPersistenceCommand::SaveAndFlush {
                snapshot,
                respond_to,
            })
            .map_err(|_| {
                io::Error::new(
                    io::ErrorKind::BrokenPipe,
                    "resources persistence writer stopped",
                )
            })?;
        Ok(response)
    }

    /// Await an acknowledgement returned by [`Self::enqueue_save_and_flush`].
    pub async fn await_save_and_flush(
        response: tokio::sync::oneshot::Receiver<io::Result<()>>,
    ) -> io::Result<()> {
        response.await.map_err(|_| {
            io::Error::new(
                io::ErrorKind::BrokenPipe,
                "resources persistence writer dropped acknowledgement",
            )
        })?
    }

    /// Replace pending snapshots, write this snapshot, and await the result.
    pub async fn save_and_flush(&self, snapshot: serde_json::Value) -> io::Result<()> {
        Self::await_save_and_flush(self.enqueue_save_and_flush(snapshot)?).await
    }

    /// `None` when this handle writes nothing.
    pub fn state_path(&self) -> Option<&std::path::Path> {
        self.state_path.as_deref()
    }

    /// Flush pending writes. Call on graceful shutdown.
    pub async fn flush(&self) {

View on GitHub (pinned to bc7f02eddd)

Solutions

Treat the result as indeterminate: verify on next startup whether the snapshot actually landed on disk.
Inspect the writer task for panics during prepare_write/replace_state_path and fix the root cause.
Retry the save once a healthy writer is available.
Ensure graceful shutdown drains in-flight saves and sends acknowledgements before stopping the writer.

Example fix

// before
ResourcesPersistence::await_save_and_flush(rx).await?; // assume durable
// after
if let Err(e) = ResourcesPersistence::await_save_and_flush(rx).await {
    if e.kind() == std::io::ErrorKind::BrokenPipe {
        tracing::error!("ack lost; snapshot durability unknown, will verify on restart");
        schedule_reconciliation();
    } else {
        return Err(e);
    }
}

Defensive patterns

Strategy: try-catch

Try / catch

match ResourcesPersistence::await_save_and_flush(rx).await {
    Ok(()) => {},
    Err(e) if e.kind() == std::io::ErrorKind::BrokenPipe => {
        // durability unknown: verify file on disk and/or retry
        if !state_file_exists_and_valid(&path).await { retry_save().await?; }
    }
    Err(e) => return Err(e.into()),
}

Prevention

Never abort the writer task while saves are pending.
Use a shutdown barrier so acknowledgements are sent before teardown.
Verify snapshot files on disk at startup to recover from lost acknowledgements.
Add tests simulating writer cancellation mid-save.

When it happens

Trigger: Calling await_save_and_flush when the writer task died between receiving the command and sending the acknowledgement — panic during serialization/replace, task abort, or runtime shutdown mid-flush.

Common situations: Process shutdown/ctrl-C racing an in-flight save; a panic in the writer's write loop; aborting the writer task in tests or during teardown while a save is pending.

Related errors

AI-assisted analysis of xai-org/grok-build@bc7f02eddd (2026-08-31). Data as JSON: /api/errors/8e28557cc95992b7. Report an issue: GitHub.