What causes "workflow fan-out failed: {err}" in tinyhumansai/openhuman?

Retry the run (resume_workflow_run) — executor faults are usually transient

workflow fan-out failed: {err}

Error message

workflow fan-out failed: {err}

What it means

Thrown by workflow_runs phase execution when the parallel fan-out itself fails. Each phase spawns its agents on tinyagents map_reduce with FailurePolicy::CollectAll and a cancellation token; TinyAgentsError::Cancelled is handled separately (persist Interrupted, terminate cleanly), so this error means the executor returned some other Err — an infrastructure failure of the parallel runner, not an individual agent failing (agent failures are carried back as data).

Source

Thrown at src/openhuman/agent/orchestration/workflow_runs/engine.rs:712

            Err(TinyAgentsError::Cancelled) => {
                log::debug!(
                    target: LOG_TARGET,
                    "[workflow_run_engine] phase.cancelled_by_sdk run={run_id} phase={}",
                    phase.name
                );
                session.abort_all().await;
                persist(
                    config,
                    &run,
                    phase_states,
                    child_run_ids,
                    WorkflowRunStatus::Interrupted,
                    None,
                    false,
                )?;
                return Ok(PhaseExecOutcome::Terminated);
            }
            Err(err) => return Err(anyhow!("workflow fan-out failed: {err}")),
        };

        let mut outcomes = Vec::with_capacity(expected_outcomes);
        for item in outcome.outcomes {
            match item.result {
                Ok(value) => outcomes.push(value),
                Err(err) => {
                    return Err(anyhow!(
                        "workflow fan-out: worker {} failed: {err}",
                        item.index
                    ));
                }
            }
        }
        if outcomes.len() != expected_outcomes {
            return Err(anyhow!(
                "workflow fan-out: expected {expected_outcomes} result(s), got {}",
                outcomes.len()

View on GitHub (pinned to a221052e0d)

Solutions

Retry the run (resume_workflow_run) — executor faults are usually transient
Lower default_concurrency / max_children on the definition to reduce parallel pressure
If reproducible, capture the inner {err} and the worker spawn/wait logs and report it as an engine bug

Example fix

// before
let run = start_workflow_run(&config, &def_id, input, None).await?; // fan-out error kills the call

// after — resume with bounded backoff on executor failure
let mut attempt = 0;
loop {
    match resume_workflow_run(&config, &run.id).await {
        Ok(r) => break r,
        Err(e) if e.to_string().contains("workflow fan-out failed") && attempt < 3 => {
            attempt += 1;
            tokio::time::sleep(Duration::from_secs(2u64.pow(attempt))).await;
        }
        Err(e) => return Err(e),
    }
}

Defensive patterns

Strategy: retry

Try / catch

let mut attempt = 0u32;
loop {
    match resume_workflow_run(&config, &run_id).await {
        Ok(r) => break Ok(r),
        Err(e) if e.to_string().contains("workflow fan-out failed") && attempt < 3 => {
            attempt += 1;
            tokio::time::sleep(Duration::from_secs(2u64.pow(attempt))).await; // backoff, then resume skips completed phases
        }
        Err(e) => break Err(e),
    }
}

Prevention

Keep definition.default_concurrency modest so map_reduce runs within resource budget
Distinguish this executor-level failure from per-agent failure before choosing retry vs fix
Always log the inner {err} — it is the only clue to the underlying executor fault

When it happens

Trigger: A panic or executor-level fault inside the map_reduce machinery during a phase with one or more agents; misconfigured concurrency (definition.default_concurrency) or an executor/runtime error propagating out of the spawned worker futures. Phases with a single agent go through the same fan-out path and can hit it too.

Common situations: A bug or panic in the orchestration session/runtime under load; resource exhaustion (too many concurrent children); version mismatch between the engine and the tinyagents executor.

Related errors

AI-assisted analysis of tinyhumansai/openhuman@a221052e0d (2026-08-16). Data as JSON: /api/errors/71793d277f7170cb. Report an issue: GitHub.