zeroclaw-labs/zeroclaw · error · TerminalPersistenceRetained

terminal persistence failed for run {}; active run and admis

Error message

terminal persistence failed for run {}; active run and admission claim remain retained: {}

What it means

persist_terminal writes a run's terminal state via store.finish_run (with a pre-computed next revision so the store's revision guard accepts it) and releases its admission claim atomically; when the store write fails, the engine wraps the StoreError in TerminalPersistenceRetained (engine.rs:127-146) and returns it, deliberately keeping the run in active_runs with its claim retained so the terminal decision can be retried instead of leaving a half-finished run. The Display text names the run_id and chains the underlying store fault via source(). It is the fail-closed counterpart to dropping state: on failure, nothing is released.

Source

Thrown at crates/zeroclaw-runtime/src/sop/engine.rs:1183

    }

    /// Persist a run that has reached a terminal state and release its claim atomically.
    fn persist_terminal(&self, run: &SopRun) -> Result<()> {
        let mut pr = PersistedRun::new(run.clone(), now_iso8601(), run.trigger_event.source);
        // The terminal write is the run's final revision; advance past the last
        // active snapshot so the store's revision guard accepts it.
        pr.revision = self.next_run_revision(&run.run_id);
        self.store.finish_run(&run.run_id, &pr).map_err(|e| {
            ::zeroclaw_log::record!(
                WARN,
                ::zeroclaw_log::Event::new(module_path!(), ::zeroclaw_log::Action::Reject)
                    .with_outcome(::zeroclaw_log::EventOutcome::Failure)
                    .with_attrs(
                        ::serde_json::json!({"run_id": run.run_id, "error": e.to_string()})
                    ),
                "SOP engine: terminal persistence failed; run and admission claim remain active"
            );
            anyhow::Error::new(TerminalPersistenceRetained {
                run_id: run.run_id.clone(),
                source: e,
            })
        })?;
        self.notify_run(run, false);
        Ok(())
    }

    /// Terminal counterpart to `persist_active_with_gate_event`: persist the
    /// terminal run, release its claim, and append the gate-resolution ledger row
    /// in one store transaction.
    fn persist_terminal_with_gate_event(&self, run: &SopRun, event: &SopEventRecord) -> Result<()> {
        let mut pr = PersistedRun::new(run.clone(), now_iso8601(), run.trigger_event.source);
        pr.revision = self.next_run_revision(&run.run_id);
        self.store
            .finish_run_with_event(&run.run_id, &pr, event)
            .map_err(|e| {
                ::zeroclaw_log::record!(

View on GitHub (pinned to 88bb9c8533)

Solutions

Read the chained source error (the {source} at the end of the message) — it names the actual store fault to fix
Restore store health (disk space, permissions, lock cleanup), then re-resolve the run: the retained active run and admission claim make the retry safe by design
Do not force-remove the run from active_runs to "clean up" — that orphans the admission claim and leaks a slot
If it recurs, capture the WARN log line "terminal persistence failed" with its run_id and error attrs for maintainers

Example fix

// before
engine.resolve_gate(request).await?; // TerminalPersistenceRetained aborts the handler outright

// after — store recovered? the run is still retained, so retry the terminal resolution
let mut last = None;
for attempt in 0..3 {
    match engine.resolve_gate(request.clone()).await {
        Ok(outcome) => { last = None; break; }
        Err(e) if e.to_string().contains("terminal persistence failed") => {
            backoff(attempt).await; // store unhealthy; give it time to recover
            last = Some(e);
        }
        Err(e) => return Err(e),
    }
}
if let Some(e) = last { return Err(e); }

Defensive patterns

Strategy: retry

Type guard

fn is_terminal_persistence_retained(e: &anyhow::Error) -> bool {
    // struct is private today; match the stable Display prefix or ask upstream for a classifier
    e.to_string().starts_with("terminal persistence failed for run ")
}

Try / catch

match engine.resolve_gate(request).await {
    Ok(outcome) => outcome,
    Err(e) if is_terminal_persistence_retained(&e) => {
        // run and claim are retained: fix store health, then re-resolve
        alert_operator(&e); schedule_retry();
    }
    Err(e) => return Err(e),
}

Prevention

Monitor store health (disk space, lock contention) where the SOP store file lives
Never manually delete active runs to clear this state — the admission claim leaks
Capture the WARN "terminal persistence failed" log line with run_id for postmortems
Keep the store on a local, writable filesystem rather than flaky network mounts

When it happens

Trigger: A run reaching a terminal Success/Fail state while store.finish_run errors — store I/O failure, a locked or corrupt store file, disk full, or a revision conflict on the terminal write.

Common situations: SQLite/store file on a full or read-only volume; store lock held by a crashed process; schema migration applied under a running engine; NFS/store path latency causing write failures.

Related errors

AI-assisted analysis of zeroclaw-labs/zeroclaw@88bb9c8533 (2026-08-23). Data as JSON: /api/errors/d89b720871c4e390. Report an issue: GitHub.