xai-org/grok-build · error · io::Error

session persistence actor stopped before durable append ackn

Error message

session persistence actor stopped before durable append acknowledgement

What it means

The AppendUpdateDurablyAndAck message was successfully sent to the persistence actor, but the oneshot response channel returned RecvError because the actor dropped the responder — i.e. the actor stopped after receiving the message but before acknowledging the durable write. Mapped to DurableAppendError::AcknowledgementLost: commit state is unknown (the write may or may not have landed).

Source

Thrown at crates/codegen/xai-grok-shell/src/session/persistence.rs:1352

        if self.noop {
            return Err(DurableAppendError::NotCommitted(io::Error::new(
                io::ErrorKind::Unsupported,
                "durable session update append is unsupported by a no-op persistence handle",
            )));
        }
        let (respond_to, response) = tokio::sync::oneshot::channel();
        self.tx
            .send(PersistenceMsg::AppendUpdateDurablyAndAck { update, respond_to })
            .map_err(|_| {
                DurableAppendError::NotCommitted(io::Error::new(
                    io::ErrorKind::BrokenPipe,
                    "session persistence actor stopped before durable append dispatch",
                ))
            })?;
        response
            .await
            .map_err(|_| {
                DurableAppendError::AcknowledgementLost(io::Error::new(
                    io::ErrorKind::BrokenPipe,
                    "session persistence actor stopped before durable append acknowledgement",
                ))
            })?
            .map_err(DurableAppendError::from)
    }
}

enum PendingAppendOutcome {
    CommittedOk(acp::SessionNotification),
    CommittedErr(acp::SessionNotification, io::Error),
    NotCommittedErr(acp::SessionNotification, io::Error),
}

struct SessionPersistence {
    info: Info,
    storage: Arc<dyn StorageAdapter>,
    /// Pending ACP notification for merging consecutive text chunks

View on GitHub (pinned to bc7f02eddd)

Solutions

Verify on disk whether the update was appended before retrying (ack-lost is ambiguous; blind retry can duplicate the entry)
Make the actor drain in-flight appends and ack them before shutting down
Wrap the actor loop so panics are caught/logged instead of silently killing the task
Keep the tokio runtime alive until pending durable appends resolve

Defensive patterns

Strategy: retry

Validate before calling

// no pre-call check can fully prevent this; ensure the actor task is healthy
if actor_join_handle.is_finished() {
    bail!("persistence actor already exited; ack would be lost");
}

Type guard

fn ack_channel_open(rx: &oneshot::Receiver<Result<(), DurableAppendError>>) -> bool {
    !rx.is_closed()
}

Try / catch

match session.append_update_durably(update).await {
    Err(DurableAppendError::AcknowledgementLost(e)) => {
        // commit state unknown: verify on disk before retrying to avoid duplicates
        if !updates_file_contains(&updates_path, &update) {
            session.append_update_durably(update).await?;
        }
    }
    Err(e) => return Err(e.into()),
    Ok(()) => {}
}

Prevention

Gracefully drain and ack pending appends before shutting down the actor
Catch panics in the actor loop so it doesn't drop responders mid-flight
Avoid Ctrl-C/abort during durable writes; use a graceful shutdown path
After ack loss, always check the updates file before re-appending to prevent duplicates

When it happens

Trigger: The response.await oneshot errors because the actor task terminated between receiving the AppendUpdateDurablyAndAck message and sending the ack — actor panic mid-fsync, task abort, or runtime shutdown during the durable write.

Common situations: Ctrl-C/shutdown racing an fsync; actor panic while persisting; abrupt process kill during append; overly aggressive session teardown that aborts the actor task before ack.

Related errors

AI-assisted analysis of xai-org/grok-build@bc7f02eddd (2026-08-31). Data as JSON: /api/errors/a866ffd0ea0d5f80. Report an issue: GitHub.