{"record":{"id":"e348a3e634304937","repo":"aaif-goose/goose","slug":"some-providers-in-skip-list-don-t-exist","errorCode":null,"errorMessage":"Some providers in skip list don't exist","messagePattern":"Some providers in skip list don't exist","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/goose-cli/src/scenario_tests/scenario_runner.rs","lineNumber":96,"sourceCode":"\n    let excluded_providers: HashSet<_> = providers_to_skip\n        .into_iter()\n        .flatten()\n        .map(|name| name.to_lowercase())\n        .collect();\n\n    let all_configs = get_provider_configs();\n\n    let all_config_len = all_configs.len();\n\n    let configs_to_test: Vec<_> = all_configs\n        .into_iter()\n        .filter(|c| !excluded_providers.contains(&c.name.to_lowercase()))\n        .collect();\n\n    if let Some(to_skip) = providers_to_skip {\n        if configs_to_test.len() != all_config_len - to_skip.len() {\n            return Err(anyhow::anyhow!(\"Some providers in skip list don't exist\"));\n        }\n    }\n\n    let mut failures = Vec::new();\n\n    for config in configs_to_test {\n        match run_provider_scenario_with_validation(\n            config,\n            test_name,\n            &message_generator,\n            &validator,\n        )\n        .await\n        {\n            Ok(_) => println!(\"✅ {} - {}\", test_name, config.name),\n            Err(e) => {\n                println!(\"❌ {} - {} FAILED: {}\", test_name, config.name, e);\n                failures.push((config.name, e));","sourceCodeStart":78,"sourceCodeEnd":114,"githubUrl":"https://github.com/aaif-goose/goose/blob/3810898a7447ec3299be72e223d3570a7aabf0ab/crates/goose-cli/src/scenario_tests/scenario_runner.rs#L78-L114","documentation":"The skip-list sanity check failed: after filtering out excluded providers, the remaining count did not equal total configs minus providers_to_skip.len(). At least one name in the skip list matched no provider (comparison is lowercased), or the list contains duplicates that inflate its len(). The check exists so silently misspelled skip entries don't hide that a provider you meant to exclude actually ran.","triggerScenarios":"Calling run_scenario with providers_to_skip = Some(&[\"opeanai\", \"REDACTED\"]) where 'opeanqi' is a typo of a real provider name; or a skip list with the same provider twice so len() over-counts the filtered set.","commonSituations":"Hand-maintained skip lists drifting from the actual provider config names; providers renamed in get_provider_configs(); duplicated entries added by merge conflicts.","solutions":["Print get_provider_configs() names and align every skip-list entry with them (case-insensitive).","Remove duplicate entries from the skip list.","Delete skip entries for providers that no longer exist in the config."],"exampleFix":"// before\nrun_scenario(\n    \"test\",\n    message_generator,\n    Some(&[\"databrickes\", \"databricks\"]), // typo + duplicate\n    validator,\n)\n\n// after\nrun_scenario(\n    \"test\",\n    message_generator,\n    Some(&[\"databricks\"]),\n    validator,\n)","handlingStrategy":"validation","validationCode":"// dedupe + validate skip list against actual config names before calling\nlet names: HashSet<String> = get_provider_configs().iter().map(|c| c.name.to_lowercase()).collect();\nlet mut skip: Vec<&str> = providers_to_skip.unwrap_or_default().to_vec();\nskip.sort_unstable();\nskip.dedup();\nlet unknown: Vec<_> = skip.iter().filter(|s| !names.contains(&s.to_lowercase())).collect();\nassert!(unknown.is_empty(), \"unknown providers in skip list: {unknown:?}\");","typeGuard":null,"tryCatchPattern":"match run_scenario(name, gen, skip, validator).await {\n    Err(e) if e.to_string().contains(\"skip list don't exist\") => {\n        // print available names, fix the list, re-run\n    }\n    r => r,\n}","preventionTips":["Derive skip lists from get_provider_configs() names, not hand-typed strings.","Deduplicate skip entries before passing them."],"tags":["testing","configuration","validation","providers"],"backgroundTag":null,"analyzedSha":"3810898a7447ec3299be72e223d3570a7aabf0ab","analyzedAt":"2026-08-16T10:14:26.282Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}