What causes "Some providers in skip list don't exist" in aaif-goose/goose?

Print get_provider_configs() names and align every skip-list entry with them (case-insensitive).

Some providers in skip list don't exist

Error message

Some providers in skip list don't exist

What it means

The skip-list sanity check failed: after filtering out excluded providers, the remaining count did not equal total configs minus providers_to_skip.len(). At least one name in the skip list matched no provider (comparison is lowercased), or the list contains duplicates that inflate its len(). The check exists so silently misspelled skip entries don't hide that a provider you meant to exclude actually ran.

Source

Thrown at crates/goose-cli/src/scenario_tests/scenario_runner.rs:96

    let excluded_providers: HashSet<_> = providers_to_skip
        .into_iter()
        .flatten()
        .map(|name| name.to_lowercase())
        .collect();

    let all_configs = get_provider_configs();

    let all_config_len = all_configs.len();

    let configs_to_test: Vec<_> = all_configs
        .into_iter()
        .filter(|c| !excluded_providers.contains(&c.name.to_lowercase()))
        .collect();

    if let Some(to_skip) = providers_to_skip {
        if configs_to_test.len() != all_config_len - to_skip.len() {
            return Err(anyhow::anyhow!("Some providers in skip list don't exist"));
        }
    }

    let mut failures = Vec::new();

    for config in configs_to_test {
        match run_provider_scenario_with_validation(
            config,
            test_name,
            &message_generator,
            &validator,
        )
        .await
        {
            Ok(_) => println!("✅ {} - {}", test_name, config.name),
            Err(e) => {
                println!("❌ {} - {} FAILED: {}", test_name, config.name, e);
                failures.push((config.name, e));

View on GitHub (pinned to 3810898a74)

Solutions

Print get_provider_configs() names and align every skip-list entry with them (case-insensitive).
Remove duplicate entries from the skip list.
Delete skip entries for providers that no longer exist in the config.

Example fix

// before
run_scenario(
    "test",
    message_generator,
    Some(&["databrickes", "databricks"]), // typo + duplicate
    validator,
)

// after
run_scenario(
    "test",
    message_generator,
    Some(&["databricks"]),
    validator,
)

Defensive patterns

Strategy: validation

Validate before calling

// dedupe + validate skip list against actual config names before calling
let names: HashSet<String> = get_provider_configs().iter().map(|c| c.name.to_lowercase()).collect();
let mut skip: Vec<&str> = providers_to_skip.unwrap_or_default().to_vec();
skip.sort_unstable();
skip.dedup();
let unknown: Vec<_> = skip.iter().filter(|s| !names.contains(&s.to_lowercase())).collect();
assert!(unknown.is_empty(), "unknown providers in skip list: {unknown:?}");

Try / catch

match run_scenario(name, gen, skip, validator).await {
    Err(e) if e.to_string().contains("skip list don't exist") => {
        // print available names, fix the list, re-run
    }
    r => r,
}

Prevention

Derive skip lists from get_provider_configs() names, not hand-typed strings.
Deduplicate skip entries before passing them.

When it happens

Trigger: Calling run_scenario with providers_to_skip = Some(&["opeanai", "REDACTED"]) where 'opeanqi' is a typo of a real provider name; or a skip list with the same provider twice so len() over-counts the filtered set.

Common situations: Hand-maintained skip lists drifting from the actual provider config names; providers renamed in get_provider_configs(); duplicated entries added by merge conflicts.

Related errors

AI-assisted analysis of aaif-goose/goose@3810898a74 (2026-08-16). Data as JSON: /api/errors/e348a3e634304937. Report an issue: GitHub.