{"record":{"id":"a881c619430ba410","repo":"alibaba/spring-ai-alibaba","slug":"all-models-failed-after-fallbackmodels-size","errorCode":null,"errorMessage":"All models failed after ${fallbackModels.size() + 1} attempts","messagePattern":"All models failed after (.+?) attempts","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"critical","filePath":"spring-ai-alibaba-agent-framework/src/main/java/com/alibaba/cloud/ai/graph/agent/interceptor/modelfallback/ModelFallbackInterceptor.java","lineNumber":100,"sourceCode":"\t\tfor (int i = 0; i < fallbackModels.size(); i++) {\n\t\t\tChatModel fallbackModel = fallbackModels.get(i);\n\t\t\ttry {\n\t\t\t\tlog.info(\"Trying fallback model {} of {}\", i + 1, fallbackModels.size());\n\n\t\t\t\t// Call the fallback model directly\n\t\t\t\tPrompt prompt = new Prompt(request.getMessages(), request.getOptions());\n\t\t\t\tvar response = fallbackModel.call(prompt);\n\n\t\t\t\treturn ModelResponse.of(response.getResult().getOutput());\n\t\t\t}\n\t\t\tcatch (Exception e) {\n\t\t\t\tlog.warn(\"Fallback model {} failed: {}\", i + 1, e.getMessage());\n\t\t\t\tlastException = e;\n\t\t\t}\n\t\t}\n\n\t\t// All models failed\n\t\tthrow new RuntimeException(\"All models failed after \" + (fallbackModels.size() + 1) + \" attempts\", lastException);\n\t}\n\n\t@Override\n\tpublic String getName() {\n\t\treturn \"ModelFallback\";\n\t}\n\n\tpublic static class Builder {\n\t\tprivate final List<ChatModel> fallbackModels = new ArrayList<>();\n\n\t\tpublic Builder addFallbackModel(ChatModel model) {\n\t\t\tthis.fallbackModels.add(model);\n\t\t\treturn this;\n\t\t}\n\n\t\tpublic Builder fallbackModels(List<ChatModel> models) {\n\t\t\tthis.fallbackModels.addAll(models);\n\t\t\treturn this;","sourceCodeStart":82,"sourceCodeEnd":118,"githubUrl":"https://github.com/alibaba/spring-ai-alibaba/blob/f82da0b50f35744c13968191be2b1cd2452ef550/spring-ai-alibaba-agent-framework/src/main/java/com/alibaba/cloud/ai/graph/agent/interceptor/modelfallback/ModelFallbackInterceptor.java#L82-L118","documentation":"ModelFallbackInterceptor tried the primary model (the intercepted handler) and every configured fallback ChatModel, and all of them threw. It exhausts the chain and rethrows a RuntimeException whose message reports the total attempt count (fallbackModels.size() + 1) with the last failure as the cause. The real root cause is in the suppressed cause chain of the last model's exception.","triggerScenarios":"Calling ModelFallbackInterceptor.interceptModel(request, handler) when handler.call(request) throws (or returns text containing 'Exception:') and every fallbackModel.call(prompt) also throws — e.g. all endpoints down, all API keys invalid/quota-exhausted, or a shared network outage.","commonSituations":"Provider-wide outages or rate limiting hitting primary and fallbacks from the same account; misconfigured base URLs or expired API keys shared across all configured models; fallback models pointing at the same unavailable cluster as the primary; regional network blocking of all provider endpoints.","solutions":["Inspect the cause chain (getCause()/lastException) of this RuntimeException to find the actual per-model failure and fix that root issue first.","Verify each fallback model's credentials, base URL, model name, and quota independently — test each ChatModel with a trivial 'ping' call at startup.","Add fallbacks from a different provider or deployment region so a single outage does not take down the whole chain.","Check the intercepted response text: if it contains 'Exception:' it is treated as failure — make sure the primary handler is not returning error text for benign reasons."],"exampleFix":"// before: all fallbacks share one key/provider\nModelFallbackInterceptor.builder()\n    .addFallbackModel(dashScopeMini)\n    .addFallbackModel(dashScopeTurbo)\n    .build();\n// after: diversify providers\nModelFallbackInterceptor.builder()\n    .addFallbackModel(openAiGpt4oMini)\n    .addFallbackModel(deepseekChat)\n    .build();","handlingStrategy":"fallback","validationCode":"List<ChatModel> fallbacks = builderModels;\nif (fallbacks.isEmpty()) throw new IllegalStateException(\"Configure at least one fallback model\");\nfallbacks.forEach(m -> {\n    try { m.call(new Prompt(\"ping\")); } catch (Exception e) { throw new IllegalStateException(\"Fallback model unhealthy: \" + e.getMessage(), e); }\n});","typeGuard":null,"tryCatchPattern":"try {\n    return agentCall();\n} catch (RuntimeException e) {\n    if (e.getMessage() != null && e.getMessage().startsWith(\"All models failed after\")) {\n        log.error(\"All models exhausted; root cause: {}\", e.getCause(), e.getCause());\n        return degradedResponse(); // serve cached/stub answer, alert ops\n    }\n    throw e;\n}","preventionTips":["Diversify fallback models across providers/regions, not just model names on one provider.","Health-check every configured ChatModel at application startup.","Monitor per-model failure rates so you see degradation before full exhaustion."],"tags":["model-call","fallback-exhausted","network","llm"],"backgroundTag":"upstream-api-error","analyzedSha":"f82da0b50f35744c13968191be2b1cd2452ef550","analyzedAt":"2026-09-09T15:32:42.421Z","contentChangedAt":"2026-09-09T15:32:42.421Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}