{"record":{"id":"9e60a127167a6c44","repo":"zeroclaw-labs/zeroclaw","slug":"all-model-providers-models-failed-after-failure","errorCode":null,"errorMessage":"All model providers/models failed after {} failure event(s). Events:","messagePattern":"All model providers/models failed after (.+?) failure event\\(s\\)\\. Events:","errorType":"exception","errorClass":"ReliableSemanticEmptyCompletion","httpStatus":null,"severity":"error","filePath":"crates/zeroclaw-providers/src/reliable.rs","lineNumber":1287,"sourceCode":"\nimpl std::error::Error for ReliableSemanticEmptyCompletion {\n    fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {\n        self.rejected_usage\n            .as_ref()\n            .map(|usage| usage as &(dyn std::error::Error + 'static))\n            .or(Some(&self.terminal_cause))\n    }\n}\n\nfn reliable_terminal_error(\n    failures: FailureEvents,\n    rejected_attempt_usage: Option<TokenUsage>,\n    final_cause_is_semantic_empty: bool,\n) -> anyhow::Error {\n    let rejected_attempt_usage = rejected_attempt_usage.or_else(accounted_rejected_attempt_usage);\n    if final_cause_is_semantic_empty {\n        let terminal_cause = zeroclaw_api::model_provider::SemanticEmptyTerminalCompletion;\n        return anyhow::Error::new(ReliableSemanticEmptyCompletion {\n            failures: failures.clone(),\n            rejected_usage: rejected_attempt_usage.map(|usage| {\n                ReliableRejectedCompletionUsage::with_terminal_cause(\n                    usage,\n                    failures,\n                    anyhow::Error::new(\n                        zeroclaw_api::model_provider::SemanticEmptyTerminalCompletion,\n                    ),\n                )\n            }),\n            terminal_cause,\n        });\n    }\n\n    match rejected_attempt_usage {\n        Some(usage) => anyhow::Error::new(ReliableRejectedCompletionUsage::new(usage, failures)),\n        None => anyhow::Error::msg(failure_aggregate(&failures)),\n    }","sourceCodeStart":1269,"sourceCodeEnd":1305,"githubUrl":"https://github.com/zeroclaw-labs/zeroclaw/blob/88bb9c8533fc57ed7a03e36ca7c9ed2bf8336dcc/crates/zeroclaw-providers/src/reliable.rs#L1269-L1305","documentation":"reliable_terminal_error builds the final error after the reliable provider chain is exhausted. Its Display (via failure_aggregate) starts with 'All model providers/models failed after N failure event(s). Events:'. When the last failure is semantically empty, it returns the typed ReliableSemanticEmptyCompletion carrying the full failure events, optional rejected-attempt usage wrapped with the SemanticEmptyTerminalCompletion terminal cause, so cost accounting and delivery layers can classify the run without guessing.","triggerScenarios":"Every candidate provider/model in the reliable chain failed and the final candidate's failure was a semantic-empty completion (empty/think-only text with no tool calls); reliable_terminal_error is called with final_cause_is_semantic_empty = true and the aggregate plus typed wrapper is produced.","commonSituations":"A chain of reasoning models all hitting max_tokens inside thinking, one misconfigured base_url replicated across chain entries, rate limits knocking out the first providers and a reasoning model failing last, all chain entries pointing at the same broken upstream.","solutions":["Read the Events list in the message: it enumerates per-provider/per-model failures; fix the dominant repeated cause first.","If events show think-only outputs, raise max_tokens or swap in one non-reasoning model as the last chain entry.","Add at least one provider on independent infrastructure so a single upstream outage cannot exhaust the chain.","Use the rejected usage attached to ReliableRejectedCompletionUsage for cost reconciliation of failed attempts."],"exampleFix":"# before: chain of reasoning models, all truncate inside thinking\nproviders = [\"deepseek-r1\", \"qwen-thinking\"]\n\n# after: end the chain with a plain chat model and generous budgets\nproviders = [\"deepseek-r1\", \"qwen-thinking\", \"gpt-4o-mini\"]\n[limits]\nmax_tokens = 4096","handlingStrategy":"fallback","validationCode":"fn chain_is_cross_vendor(models: &[(&str, &str)]) -> bool {\n    models.iter().map(|(p, _)| *p).collect::<std::collections::HashSet<_>>().len() > 1\n}","typeGuard":null,"tryCatchPattern":"match reliable.chat(request, None).await {\n    Ok(resp) => Ok(resp),\n    Err(e) => {\n        if e.downcast_ref::<ReliableSemanticEmptyCompletion>().is_some() {\n            log::warn!(\"chain exhausted with semantic-empty terminal: {e:#}\");\n        }\n        escalate_to_operator(&e) // chain already retried everything; do not blind-retry\n    }\n}","preventionTips":["Configure at least two vendors in the reliable chain so one outage or one reasoning-model quirk cannot exhaust it.","Review failure Events in the aggregate before changing code; the dominant repeated cause is the fix target.","End every chain with a plain chat model to convert think-only terminals into successes."],"tags":["llm","provider-chain","reliable","aggregate","fallback-exhausted"],"backgroundTag":"llm-provider-chain-exhausted","analyzedSha":"88bb9c8533fc57ed7a03e36ca7c9ed2bf8336dcc","analyzedAt":"2026-08-23T01:07:41.857Z","schemaVersion":2},"datasetVersion":"2026-08-23T08:06:27.607Z"}