{"record":{"id":"71793d277f7170cb","repo":"tinyhumansai/openhuman","slug":"workflow-fan-out-failed-err","errorCode":null,"errorMessage":"workflow fan-out failed: {err}","messagePattern":"workflow fan-out failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/openhuman/agent/orchestration/workflow_runs/engine.rs","lineNumber":712,"sourceCode":"            Err(TinyAgentsError::Cancelled) => {\n                log::debug!(\n                    target: LOG_TARGET,\n                    \"[workflow_run_engine] phase.cancelled_by_sdk run={run_id} phase={}\",\n                    phase.name\n                );\n                session.abort_all().await;\n                persist(\n                    config,\n                    &run,\n                    phase_states,\n                    child_run_ids,\n                    WorkflowRunStatus::Interrupted,\n                    None,\n                    false,\n                )?;\n                return Ok(PhaseExecOutcome::Terminated);\n            }\n            Err(err) => return Err(anyhow!(\"workflow fan-out failed: {err}\")),\n        };\n\n        let mut outcomes = Vec::with_capacity(expected_outcomes);\n        for item in outcome.outcomes {\n            match item.result {\n                Ok(value) => outcomes.push(value),\n                Err(err) => {\n                    return Err(anyhow!(\n                        \"workflow fan-out: worker {} failed: {err}\",\n                        item.index\n                    ));\n                }\n            }\n        }\n        if outcomes.len() != expected_outcomes {\n            return Err(anyhow!(\n                \"workflow fan-out: expected {expected_outcomes} result(s), got {}\",\n                outcomes.len()","sourceCodeStart":694,"sourceCodeEnd":730,"githubUrl":"https://github.com/tinyhumansai/openhuman/blob/a221052e0df5b1f7598fceba7329fd1af95d6699/src/openhuman/agent/orchestration/workflow_runs/engine.rs#L694-L730","documentation":"Thrown by workflow_runs phase execution when the parallel fan-out itself fails. Each phase spawns its agents on tinyagents map_reduce with FailurePolicy::CollectAll and a cancellation token; TinyAgentsError::Cancelled is handled separately (persist Interrupted, terminate cleanly), so this error means the executor returned some other Err — an infrastructure failure of the parallel runner, not an individual agent failing (agent failures are carried back as data).","triggerScenarios":"A panic or executor-level fault inside the map_reduce machinery during a phase with one or more agents; misconfigured concurrency (definition.default_concurrency) or an executor/runtime error propagating out of the spawned worker futures. Phases with a single agent go through the same fan-out path and can hit it too.","commonSituations":"A bug or panic in the orchestration session/runtime under load; resource exhaustion (too many concurrent children); version mismatch between the engine and the tinyagents executor.","solutions":["Retry the run (resume_workflow_run) — executor faults are usually transient","Lower default_concurrency / max_children on the definition to reduce parallel pressure","If reproducible, capture the inner {err} and the worker spawn/wait logs and report it as an engine bug"],"exampleFix":"// before\nlet run = start_workflow_run(&config, &def_id, input, None).await?; // fan-out error kills the call\n\n// after — resume with bounded backoff on executor failure\nlet mut attempt = 0;\nloop {\n    match resume_workflow_run(&config, &run.id).await {\n        Ok(r) => break r,\n        Err(e) if e.to_string().contains(\"workflow fan-out failed\") && attempt < 3 => {\n            attempt += 1;\n            tokio::time::sleep(Duration::from_secs(2u64.pow(attempt))).await;\n        }\n        Err(e) => return Err(e),\n    }\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"let mut attempt = 0u32;\nloop {\n    match resume_workflow_run(&config, &run_id).await {\n        Ok(r) => break Ok(r),\n        Err(e) if e.to_string().contains(\"workflow fan-out failed\") && attempt < 3 => {\n            attempt += 1;\n            tokio::time::sleep(Duration::from_secs(2u64.pow(attempt))).await; // backoff, then resume skips completed phases\n        }\n        Err(e) => break Err(e),\n    }\n}","preventionTips":["Keep definition.default_concurrency modest so map_reduce runs within resource budget","Distinguish this executor-level failure from per-agent failure before choosing retry vs fix","Always log the inner {err} — it is the only clue to the underlying executor fault"],"tags":["workflow-runs","fan-out","executor","retry"],"backgroundTag":null,"analyzedSha":"a221052e0df5b1f7598fceba7329fd1af95d6699","analyzedAt":"2026-08-16T12:47:06.542Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}