{"record":{"id":"558892638f52706c","repo":"aaif-goose/goose","slug":"test-failed-for-provider-s","errorCode":null,"errorMessage":"Test '{}' failed for {} provider(s)","messagePattern":"Test '(.+?)' failed for (.+?) provider\\(s\\)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/goose-cli/src/scenario_tests/scenario_runner.rs","lineNumber":124,"sourceCode":"            &message_generator,\n            &validator,\n        )\n        .await\n        {\n            Ok(_) => println!(\"✅ {} - {}\", test_name, config.name),\n            Err(e) => {\n                println!(\"❌ {} - {} FAILED: {}\", test_name, config.name, e);\n                failures.push((config.name, e));\n            }\n        }\n    }\n\n    if !failures.is_empty() {\n        println!(\"\\n=== Test Failures for {} ===\", test_name);\n        for (provider, error) in &failures {\n            println!(\"❌ {}: {}\", provider, error);\n        }\n        return Err(anyhow::anyhow!(\n            \"Test '{}' failed for {} provider(s)\",\n            test_name,\n            failures.len()\n        ));\n    }\n\n    Ok(())\n}\n\nasync fn run_provider_scenario_with_validation<F>(\n    config: &ProviderConfig,\n    test_name: &str,\n    message_generator: &MessageGenerator<'_>,\n    validator: &F,\n) -> Result<()>\nwhere\n    F: Fn(&ScenarioResult) -> Result<()>,\n{","sourceCodeStart":106,"sourceCodeEnd":142,"githubUrl":"https://github.com/aaif-goose/goose/blob/3810898a7447ec3299be72e223d3570a7aabf0ab/crates/goose-cli/src/scenario_tests/scenario_runner.rs#L106-L142","documentation":"Aggregate failure after a scenario ran across every non-skipped provider: at least one provider's run or validator returned Err, so run_scenario collects the failures and raises this summary. The per-provider details (provider name + error) were already printed as '❌ name: error' lines just above; this error only carries the count.","triggerScenarios":"Any run_provider_scenario_with_validation failure across the matrix: a provider API error, a replay/record mismatch, or a validator assertion rejecting the output for one provider while others passed.","commonSituations":"One flaky provider (rate limits, expired key) failing an otherwise green matrix; model-specific output differences breaking validators tuned to another model's phrasing.","solutions":["Scroll up to the '=== Test Failures ===' block and read each '❌ provider: error' line to identify which provider and why.","Fix the root cause for that provider (credentials, re-record its replay file, or relax/fix the validator for its output style).","If the provider is known-broken, add it to providers_to_skip for the run."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"let failures = run_all_providers(&scenario).await;\nif let Err(agg) = failures {\n    // parse the per-provider ❌ lines already printed; fail CI with the first root cause\n    if agg.to_string().contains(\"failed for 1 provider\") {\n        eprintln!(\"single-provider failure — likely env/credential specific\");\n    }\n    return Err(agg);\n}","preventionTips":["Treat the printed per-provider failure lines as the real signal; the summary only counts.","Keep per-provider credentials fresh so env issues don't masquerade as test failures."],"tags":["testing","scenarios","aggregation","multi-provider"],"backgroundTag":null,"analyzedSha":"3810898a7447ec3299be72e223d3570a7aabf0ab","analyzedAt":"2026-08-16T10:14:26.282Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}