aaif-goose/goose · error · ValueError

--model must be in provider/model form, e.g. anthropic/claud

Error message

--model must be in provider/model form, e.g. anthropic/claude-sonnet-4-6

What it means

Raised by build_harbor_config in the harbor benchmark runner when the --model string contains no '/'. The runner requires provider/model form (e.g. anthropic/claude-sonnet-4-6) because it splits on the first slash to get the provider prefix, which it then uses to look up required secrets in PROVIDER_SECRETS and to configure the agent. A bare model name gives it no provider, so it refuses up front rather than failing later with a cryptic provider error.

Source

Thrown at evals/harbor/runner.py:128

    if index_url is None:
        return {}
    return {key: index_url for key in PACKAGE_INDEX_ENV_VARS}


def dataset_config(dataset_ref: str, tasks: list[str]) -> dict[str, Any]:
    name, sep, ref = dataset_ref.rpartition("@")
    dataset_name = name if sep else dataset_ref
    dataset: dict[str, Any] = {"name": dataset_name}
    if sep:
        dataset["ref" if "/" in name else "version"] = ref
    if tasks:
        dataset["task_names"] = tasks
    return dataset


def build_harbor_config(args: argparse.Namespace) -> dict[str, Any]:
    if "/" not in args.model:
        raise ValueError("--model must be in provider/model form, e.g. anthropic/claude-sonnet-4-6")
    if args.trials < 1:
        raise ValueError("--trials must be at least 1")
    if args.concurrency < 1:
        raise ValueError("--concurrency must be at least 1")
    if args.timeout_multiplier <= 0:
        raise ValueError("--timeout-multiplier must be positive")

    goose_binary = args.goose_binary.expanduser().resolve()
    if not goose_binary.is_file():
        raise ValueError(f"--goose-binary does not exist or is not a file: {args.goose_binary}")

    config_yaml, extension_entries = render_goose_config(args.extensions)

    provider = args.model.split("/", 1)[0]
    missing_secrets = [
        key for key in PROVIDER_SECRETS.get(provider, []) if not os.environ.get(key)
    ]
    if missing_secrets:

View on GitHub (pinned to 3810898a74)

Solutions

Pass the provider-qualified id: --model anthropic/claude-sonnet-4-6
Check the provider prefix matches one PROVIDER_SECRETS knows (anthropic, openai, ...) so the secret lookup in error 53 does not silently find nothing
If wrapping the runner in scripts, validate model strings with a '/' check before invoking it

Example fix

# before
python runner.py --model claude-sonnet-4-6

# after
python runner.py --model anthropic/claude-sonnet-4-6

Defensive patterns

Strategy: validation

Validate before calling

def is_provider_model(model: str) -> bool:
    provider, sep, name = model.partition('/')
    return bool(sep) and bool(provider) and bool(name)

assert is_provider_model(args.model), '--model must be provider/model, e.g. anthropic/claude-sonnet-4-6'

Prevention

Always copy model ids as provider/model from provider docs or leaderboards
Validate the slash form in wrapper scripts before invoking the runner
Keep the provider prefix consistent with the secrets map so later env checks are meaningful

When it happens

Trigger: Running with --model claude-sonnet-4-6; --model gpt-4o; any model id copied without its provider prefix.

Common situations: Copying the model id column of a leaderboard rather than the provider/model column; switching from another harness that takes bare model names; defaulting mentally to just the model because the provider is implied by the API key in use.

Related errors

AI-assisted analysis of aaif-goose/goose@3810898a74 (2026-08-16). Data as JSON: /api/errors/eed3bc8b19d92697. Report an issue: GitHub.