aaif-goose/goose · error

Test '{}' failed for {} provider(s)

Error message

Test '{}' failed for {} provider(s)

What it means

Aggregate failure after a scenario ran across every non-skipped provider: at least one provider's run or validator returned Err, so run_scenario collects the failures and raises this summary. The per-provider details (provider name + error) were already printed as '❌ name: error' lines just above; this error only carries the count.

Source

Thrown at crates/goose-cli/src/scenario_tests/scenario_runner.rs:124

            &message_generator,
            &validator,
        )
        .await
        {
            Ok(_) => println!("✅ {} - {}", test_name, config.name),
            Err(e) => {
                println!("❌ {} - {} FAILED: {}", test_name, config.name, e);
                failures.push((config.name, e));
            }
        }
    }

    if !failures.is_empty() {
        println!("\n=== Test Failures for {} ===", test_name);
        for (provider, error) in &failures {
            println!("❌ {}: {}", provider, error);
        }
        return Err(anyhow::anyhow!(
            "Test '{}' failed for {} provider(s)",
            test_name,
            failures.len()
        ));
    }

    Ok(())
}

async fn run_provider_scenario_with_validation<F>(
    config: &ProviderConfig,
    test_name: &str,
    message_generator: &MessageGenerator<'_>,
    validator: &F,
) -> Result<()>
where
    F: Fn(&ScenarioResult) -> Result<()>,
{

View on GitHub (pinned to 3810898a74)

Solutions

1. Scroll up to the '=== Test Failures ===' block and read each '❌ provider: error' line to identify which provider and why.
2. Fix the root cause for that provider (credentials, re-record its replay file, or relax/fix the validator for its output style).
3. If the provider is known-broken, add it to providers_to_skip for the run.

let failures = run_all_providers(&scenario).await;
if let Err(agg) = failures {
    // parse the per-provider ❌ lines already printed; fail CI with the first root cause
    if agg.to_string().contains("failed for 1 provider") {
        eprintln!("single-provider failure — likely env/credential specific");
    }
    return Err(agg);
}

When it happens

Trigger: Any run_provider_scenario_with_validation failure across the matrix: a provider API error, a replay/record mismatch, or a validator assertion rejecting the output for one provider while others passed.

Common situations: One flaky provider (rate limits, expired key) failing an otherwise green matrix; model-specific output differences breaking validators tuned to another model's phrasing.

Related errors

• No messages found in scenario result
• Provider '{}' not found. Available: {}
• Some providers in skip list don't exist
• Test replay failed for '{}' ({}): {}. File deleted - re-run
• Test replay failed for '{}' ({}) - missing recorded interact

AI-assisted analysis of aaif-goose/goose@3810898a74 (2026-08-16). Data as JSON: /api/errors/558892638f52706c. Report an issue: GitHub.