tinyhumansai/openhuman · critical

workflow run {run_id} vanished mid-phase

Error message

workflow run {run_id} vanished mid-phase

What it means

Thrown inside workflow_runs::engine phase execution when the run row — reloaded so the engine mutates the latest phase_states projection — is suddenly missing from the ledger. Same failure class as 'vanished mid-loop', detected at phase start instead of between phases: the run was deleted by something outside the engine while it held a live run id.

Source

Thrown at src/openhuman/agent/orchestration/workflow_runs/engine.rs:515

/// cancellation landed mid-phase (the terminal status is persisted first).
#[allow(clippy::too_many_arguments)]
pub(super) async fn execute_phase(
    config: &Config,
    run_id: &str,
    definition: &WorkflowDefinition,
    session: &crate::openhuman::agent::orchestration::AgentOrchestrationSession,
    cancel: &Arc<AtomicBool>,
    model_override: Option<String>,
    phase: &WorkflowPhase,
    total_spawned: u32,
) -> Result<PhaseExecOutcome> {
    use crate::openhuman::agent::orchestration::{
        AgentStatus, SpawnAgentRequest, WaitAgentOptions,
    };

    // Reload so the phase state we mutate + persist is the latest projection.
    let run = get_workflow_run(&config.workspace_dir, run_id)?
        .ok_or_else(|| anyhow!("workflow run {run_id} vanished mid-phase"))?;
    let mut phase_states = run.phase_states.clone();
    let mut child_run_ids = run.child_run_ids.clone();
    // Children launched *this* phase (the reducer delta added to `total_spawned`).
    let mut spawned_this_phase: u32 = 0;

    log::debug!(
        target: LOG_TARGET,
        "[workflow_run_engine] phase.start run={run_id} phase={} agents={} spawned_so_far={}",
        phase.name,
        phase.agent_ids.len(),
        total_spawned
    );
    set_phase_status(&mut phase_states, &phase.name, PHASE_RUNNING, None);
    persist(
        config,
        &run,
        phase_states.clone(),
        child_run_ids.clone(),

View on GitHub (pinned to a221052e0d)

Solutions

1. Make run deletion exclusive: refuse to delete runs whose status is Running, or stop the run first via cancel
2. Coordinate workspace resets with the engine (drain runs before resetting)
3. Recover by starting a new run with the same input once the interference is gone

match execute_phase(config, run_id, session, cancel, model, phase, spawned).await {
    Err(e) if e.to_string().contains("vanished mid-phase") => {
        // external deletion during execution — abort the loop, report interference
        abort_and_report_data_loss(run_id, e)
    }
    other => other,
}

When it happens

Trigger: The run ledger row is deleted (workspace wipe, DB file removal, concurrent delete call) after phase selection succeeded but before/during execute_phase's reload. Long phases widen the window; any external ledger mutation during execution triggers it.

Common situations: Workspace cleanup jobs with no awareness of running workflows; running integration tests against a workspace a daemon also cleans; deleting runs from a UI to 'tidy up' while they execute.

Related errors

• workflow run {run_id} vanished mid-loop
• team missing after creation: {team_id}
• workflow fan-out: expected {expected_outcomes} result(s), go
• whatsapp_data write lock poisoned: {e}
• AgentDefinitionRegistry missing after init

AI-assisted analysis of tinyhumansai/openhuman@a221052e0d (2026-08-16). Data as JSON: /api/errors/1733d83c66204a47. Report an issue: GitHub.