tinyhumansai/openhuman · critical

workflow run {run_id} vanished mid-loop

Error message

workflow run {run_id} vanished mid-loop

What it means

Thrown inside workflow_runs::engine::select_next_phase when the engine loop reloads the run row between phases and get_workflow_run returns None. The run existed when the loop started (its id came from start/resume) but its ledger row was deleted mid-execution — external interference with the run ledger, not a caller-input problem.

Source

Thrown at src/openhuman/agent/orchestration/workflow_runs/engine.rs:428

    Continue { spawned: u32 },
    /// The run reached a terminal status (already persisted) — route to `done`.
    Terminated,
}

/// `dispatch` step: reload the run, honour cancellation, and pick the next
/// runnable phase (pending, all deps `completed`). When none remains, persist the
/// terminal status (Completed / Failed) and return [`PhaseSelection::Terminated`].
pub(super) async fn select_next_phase(
    config: &Config,
    run_id: &str,
    definition: &WorkflowDefinition,
    cancel: &Arc<AtomicBool>,
    session: &crate::openhuman::agent::orchestration::AgentOrchestrationSession,
) -> Result<PhaseSelection> {
    // Reload so we read the latest phase_states (and a resume picks up persisted
    // progress).
    let run = get_workflow_run(&config.workspace_dir, run_id)?
        .ok_or_else(|| anyhow!("workflow run {run_id} vanished mid-loop"))?;
    let phase_states = run.phase_states.clone();
    let child_run_ids = run.child_run_ids.clone();

    // Cancellation check between phases.
    if cancel.load(Ordering::SeqCst) {
        log::debug!(
            target: LOG_TARGET,
            "[workflow_run_engine] loop.cancelled run={run_id}"
        );
        session.abort_all().await;
        persist(
            config,
            &run,
            phase_states,
            child_run_ids,
            WorkflowRunStatus::Interrupted,
            None,
            false,

View on GitHub (pinned to a221052e0d)

Solutions

1. Stop deleting/resetting the workspace while runs are in flight — gate resets on no active runs
2. If the deletion was accidental, restart the workflow as a new run (the old one is unrecoverable)
3. Audit for concurrent processes writing the same run-ledger DB

// in the engine loop caller — the row can vanish at any await point, so catch and classify
let result = run_engine_loop(config, run_id).await;
match result {
    Err(e) if e.to_string().contains("vanished mid-loop") => {
        log::error!("run {run_id} deleted mid-execution by an external writer");
        // surface as data-loss/interference, not a workflow failure; do not blind-retry
        Err(e)
    }
    other => other,
}

When it happens

Trigger: A workflow run is executing while something deletes its row: workspace reset/wipe, ledger DB deleted or restored, or a concurrent delete-RPC/tool acting on the same run. The engine aborts at the next phase boundary with this error.

Common situations: Test harness resetting the workspace while a long workflow runs; two operators (or a script plus UI) managing the same workspace; backup-restore of the ledger underneath a live run.

Related errors

• workflow run {run_id} vanished mid-phase
• team missing after creation: {team_id}
• workflow fan-out: expected {expected_outcomes} result(s), go
• whatsapp_data write lock poisoned: {e}
• AgentDefinitionRegistry missing after init

AI-assisted analysis of tinyhumansai/openhuman@a221052e0d (2026-08-16). Data as JSON: /api/errors/f653258421fbb21d. Report an issue: GitHub.