xai-org/grok-build · error · SchedulerError::Persistence

resources persistence writer dropped acknowledgement

Error message

resources persistence writer dropped acknowledgement

What it means

In the scheduler actor's fire_next_task expiry persistence path, when the oneshot channel carrying the writer's acknowledgement is itself dropped without a reply (JoinError on the writer task), the outcome is marked Unknown with a BrokenPipe io::Error "resources persistence writer dropped acknowledgement". This means it cannot be determined whether the expiry was durably persisted — the writer task died or was cancelled mid-operation.

Source

Thrown at crates/codegen/xai-grok-tools/src/implementations/grok_build/scheduler/actor.rs:353

                .enqueue_save_and_flush(res.serialize());
            drop(res);
            tracing::info!(task_id = %task_id, "Scheduled task expired; removing without firing");

            let persistence = match acknowledgement {
                Ok(acknowledgement) => {
                    let deadline = tokio::time::Instant::now() + DURABILITY_BARRIER_TIMEOUT;
                    tokio::select! {
                        _ = self.cancel_token.cancelled() => {
                            ExpiryPersistenceOutcome::Unknown(SchedulerError::Cancelled)
                        }
                        result = tokio::time::timeout_at(deadline, acknowledgement) => {
                            match result {
                                Ok(Ok(Ok(()))) => ExpiryPersistenceOutcome::Committed,
                                Ok(Ok(Err(error))) => {
                                    ExpiryPersistenceOutcome::NotCommitted(error)
                                }
                                Ok(Err(_)) => ExpiryPersistenceOutcome::Unknown(
                                    SchedulerError::Persistence(std::io::Error::new(
                                        std::io::ErrorKind::BrokenPipe,
                                        "resources persistence writer dropped acknowledgement",
                                    )),
                                ),
                                Err(_) => {
                                    ExpiryPersistenceOutcome::Unknown(SchedulerError::Timeout)
                                }
                            }
                        }
                    }
                }
                Err(error) => ExpiryPersistenceOutcome::NotCommitted(error),
            };
            match persistence {
                ExpiryPersistenceOutcome::Committed => {}
                ExpiryPersistenceOutcome::NotCommitted(error) => {
                    tracing::warn!(
                        %task_id,

View on GitHub (pinned to bc7f02eddd)

Solutions

Treat the outcome as Unknown and reconcile on startup: re-derive task absence/presence from disk rather than assuming the write failed.
Inspect writer task logs for a panic/cancel and fix the underlying I/O error that killed it.
Retry the save via a fresh writer instance once it is restarted.
Ensure the writer task is not aborted while save requests are outstanding (drain before shutdown).

Example fix

// before
let outcome = scheduler.fire_next_task(...).await?; // assume committed
// after
if let ExpiryPersistenceOutcome::Unknown(err) = outcome {
    tracing::error!("expiry persistence indeterminate: {err}; will reconcile from disk");
    self.reconcile_from_disk().await?;
}

Defensive patterns

Strategy: retry

Validate before calling

// Before relying on persisted state, confirm the writer task is alive:
let writer_alive = writer_handle.is_finished() == false;

Try / catch

match outcome {
    ExpiryPersistenceOutcome::Committed => {/* ok */},
    ExpiryPersistenceOutcome::Unknown(err) => {
        tracing::error!("indeterminate: {err}");
        retry_with_backoff(|| persist_expiry(absence)).await?;
        reconcile_from_disk().await?;
    }
    ExpiryPersistenceOutcome::NotCommitted(e) => return Err(e.into()),
}

Prevention

Drain in-flight saves before stopping the writer task.
Catch panics inside the writer loop so one bad write does not kill the task.
Reconcile durability from disk on startup after any Unknown outcome.
Monitor writer task liveness with a health check.

When it happens

Trigger: The persistence writer task panics or is aborted while handling a SaveAndFlush command, so the oneshot Sender is dropped without sending; the runtime shuts down the writer task before the acknowledgement round-trips.

Common situations: Runtime shutdown during graceful stop while expiry persistence is in flight; a panic inside the writer's disk I/O loop; the background writer task being cancelled by a timeout harness (as in the durable-persistence tests).

Related errors

AI-assisted analysis of xai-org/grok-build@bc7f02eddd (2026-08-31). Data as JSON: /api/errors/54743dd3d436d7d8. Report an issue: GitHub.