aaif-goose/goose · error · ValueError

--trials must be at least 1

Error message

--trials must be at least 1

What it means

Raised by build_harbor_config in the harbor benchmark runner when --trials is less than 1. Trials is the number of repetitions per task; zero or negative trials would produce empty result sets, so the runner rejects the value immediately. Pure argument validation before any config is rendered or any job submitted.

Solutions

1. Use --trials 1 for the smallest valid run (one pass per task)
2. If trials came from a script, guard the value: only invoke the runner when trials >= 1
3. For pure config validation without runs, rely on a dry validation path rather than zeroing trials

assert args.trials >= 1, f'--trials must be at least 1, got {args.trials}'

When it happens

Trigger: Running with --trials 0; a negative value like --trials -2; a shell loop that computes trials and can yield 0.

Common situations: Smoke-testing with '--trials 0' intending 'run nothing, just validate'; arithmetic on dataset size producing 0; typoing an intended --trials 10 as part of another flag.

Related errors

• --concurrency must be at least 1
• Job name must start with a letter or number and contain…
• --model must be in provider/model form, e.g…
• --timeout-multiplier must be positive
• --goose-binary does not exist or is not a file

AI-assisted analysis of aaif-goose/goose@3810898a74 (2026-08-16). Data as JSON: /api/errors/18b2f6e88e7816d8. Report an issue: GitHub.

Appendix: source

Thrown at evals/harbor/runner.py:130

    return {key: index_url for key in PACKAGE_INDEX_ENV_VARS}


def dataset_config(dataset_ref: str, tasks: list[str]) -> dict[str, Any]:
    name, sep, ref = dataset_ref.rpartition("@")
    dataset_name = name if sep else dataset_ref
    dataset: dict[str, Any] = {"name": dataset_name}
    if sep:
        dataset["ref" if "/" in name else "version"] = ref
    if tasks:
        dataset["task_names"] = tasks
    return dataset


def build_harbor_config(args: argparse.Namespace) -> dict[str, Any]:
    if "/" not in args.model:
        raise ValueError("--model must be in provider/model form, e.g. anthropic/claude-sonnet-4-6")
    if args.trials < 1:
        raise ValueError("--trials must be at least 1")
    if args.concurrency < 1:
        raise ValueError("--concurrency must be at least 1")
    if args.timeout_multiplier <= 0:
        raise ValueError("--timeout-multiplier must be positive")

    goose_binary = args.goose_binary.expanduser().resolve()
    if not goose_binary.is_file():
        raise ValueError(f"--goose-binary does not exist or is not a file: {args.goose_binary}")

    config_yaml, extension_entries = render_goose_config(args.extensions)

    provider = args.model.split("/", 1)[0]
    missing_secrets = [
        key for key in PROVIDER_SECRETS.get(provider, []) if not os.environ.get(key)
    ]
    if missing_secrets:
        raise ValueError(
            f"Missing env vars for provider '{provider}': {', '.join(missing_secrets)}. "

View on GitHub (pinned to 3810898a74)