zeroclaw-labs/zeroclaw · error · anyhow::Error

Run {} cannot resume: its parked checkpoint snapshot is not

Error message

Run {} cannot resume: its parked checkpoint snapshot is not yet durably persisted (retrying)

What it means

Resume refuses to proceed while is_park_persist_pending(run_id) is true: the run's exec claim predates this attempt, and reacquiring on top of an unpersisted park would let a later rollback or maintenance retry confuse 'freshly reacquired' with 'pre-existing, must survive'. The guard fails closed; a maintenance tick's retry durably persists the park, after which resume works.

Source

Thrown at crates/zeroclaw-runtime/src/sop/engine.rs:4333

                let run_id = state.run_id.clone();
                ::zeroclaw_log::record!(
                    WARN,
                    ::zeroclaw_log::Event::new(module_path!(), ::zeroclaw_log::Action::Reject)
                        .with_outcome(::zeroclaw_log::EventOutcome::Failure)
                        .with_attrs(::serde_json::json!({"run_id": run_id})),
                    "SOP engine: active run not found"
                );
                bail!("Active run not found: {}", state.run_id);
            }
        };

        // Refuse to resume while the checkpoint's parked snapshot has not yet
        // been durably persisted (see `is_park_persist_pending`'s doc): the kept
        // claim predates this attempt, and reacquiring on top of it would give a
        // later rollback or a maintenance retry no way to distinguish "freshly
        // reacquired" from "pre-existing, must survive."
        if self.is_park_persist_pending(&state.run_id) {
            bail!(
                "Run {} cannot resume: its parked checkpoint snapshot is not yet durably persisted (retrying)",
                state.run_id
            );
        }

        let sop = self
            .sops
            .iter()
            .find(|s| s.name == sop_name)
            .ok_or_else(|| {
                ::zeroclaw_log::record!(
                    WARN,
                    ::zeroclaw_log::Event::new(module_path!(), ::zeroclaw_log::Action::Fail)
                        .with_outcome(::zeroclaw_log::EventOutcome::Failure)
                        .with_attrs(::serde_json::json!({"sop_name": sop_name.as_str()})),
                    "SOP engine: sop no longer loaded (definition removed mid-run)"
                );
                anyhow::Error::msg(format!("SOP '{sop_name}' no longer loaded"))

View on GitHub (pinned to 88bb9c8533)

Solutions

Retry resume with backoff; the condition is transient and clears once the park is durably persisted.
If retries keep failing, inspect durable-store health and engine logs for park-persist retry failures.
Do not strip or work around the guard; it protects claim accounting during the pending window.

Example fix

// before: immediate resume after seeing PausedCheckpoint
engine.resume_checkpoint(state).await?; // trips while park persist is in flight

// after: bounded retry until the snapshot is durable
let mut attempt = 0u32;
loop {
    match engine.resume_checkpoint(state.clone()).await {
        Ok(out) => break out,
        Err(e) if e.to_string().contains("not yet durably persisted") && attempt < 8 => {
            attempt += 1;
            tokio::time::sleep(std::time::Duration::from_millis(250 * u64::from(attempt))).await;
        }
        Err(e) => return Err(e),
    }
}

Defensive patterns

Strategy: retry

Try / catch

let mut attempt = 0u32;
loop {
    match engine.resume_checkpoint(state.clone()).await {
        Ok(out) => break out,
        Err(e) if e.to_string().contains("not yet durably persisted") && attempt < 8 => {
            attempt += 1;
            tokio::time::sleep(std::time::Duration::from_millis(250 * u64::from(attempt))).await;
        }
        Err(e) => return Err(e),
    }
}

Prevention

Delay resume slightly after observing PausedCheckpoint so the park persist can land.
Watch for park-persist retry failures in logs when resumes keep bouncing.
Never acquire claims manually to work around the pending window.

When it happens

Trigger: Resuming immediately after a run parks at a checkpoint while its park snapshot write is still outstanding (slow disk, store latency, or a failed persist being retried).

Common situations: Automated resume pipelines reacting instantly to a PausedCheckpoint status; degraded durable storage slowing the park write; heavy fsync load delaying the snapshot.

Related errors

AI-assisted analysis of zeroclaw-labs/zeroclaw@88bb9c8533 (2026-08-23). Data as JSON: /api/errors/44f30f10b3036f1f. Report an issue: GitHub.