headroomlabs-ai/headroom · error · ValueError

HeadroomSuite requires at least one scenario

Error message

HeadroomSuite requires at least one scenario

What it means

Raised by HeadroomSuite._require_scenarios(), a guard invoked by suite operations (run, manifest writing, etc.) when the suite's internal scenario list is empty. The suite cannot do anything meaningful with zero scenarios, so it fails fast with a clear message instead of producing empty reports.

Source

Thrown at headroom/testing/harness.py:1094

            benchmark_ref=benchmark_ref,
            provider=provider,
            now=now,
        )
        written: list[Path] = []
        for scenario_name, manifest in manifests.items():
            target = target_dir / f"{scenario_name}.agent-evals.json"
            target.write_text(
                json.dumps(manifest.to_dict(), indent=2, sort_keys=True) + "\n",
                encoding="utf-8",
            )
            written.append(target)
        return tuple(written)

    WriteAgentEvalsManifests = write_agent_evals_manifests

    def _require_scenarios(self) -> tuple[HarnessScenario, ...]:
        if not self._scenarios:
            raise ValueError("HeadroomSuite requires at least one scenario")
        return tuple(self._scenarios)


@dataclass(frozen=True)
class HarnessScenario:
    """A fully-built, immutable Headroom test scenario."""

    name: str
    provider: ProviderTarget
    platform: PlatformTarget
    headroom_config: HeadroomConfig
    proxy_config: ProxyConfig
    metadata: dict[str, Any]

    @property
    def contract(self) -> ScenarioContract:
        return ScenarioContract(
            headroom_fields=_field_contract("headroom", HeadroomConfig),

View on GitHub (pinned to 322425c43b)

Solutions

1. Add at least one scenario before running: `suite.add(HarnessScenario(...))` or `suite.add(headroom_builder)`.
2. If scenarios come from a dynamic list, assert or warn when it is empty before constructing the suite.
3. Check the filter/config that produces the scenario list (env var, file glob, model allowlist).

Example fix

# before
suite = HeadroomSuite()
report = suite.run()  # ValueError

# after
suite = HeadroomSuite().add(build_baseline())
report = suite.run()

if not suite.scenarios:
    raise RuntimeError("no scenarios configured; check suite setup")
suite.run()

When it happens

Trigger: Creating HeadroomSuite() and immediately calling run(), write_agent_evals_manifests(...), or any operation that routes through _require_scenarios() without ever calling add()/extend(); a filter step that removes all scenarios before running.

Common situations: Suite assembled conditionally from a config-driven list that ends up empty (bad filter, wrong env var, empty glob); CI parameter sets that select zero scenarios; refactoring that moved add() calls behind a flag that is off.

Related errors

• mode must be 'token' or 'cache'
• ScenarioOrchestrator requires at least one scenario
• ScenarioOrchestrator.run requires at least one task
• {name} is read-only; mutate its fields instead
• unknown Headroom harness config field: {name}

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/4e983872451d4c9a. Report an issue: GitHub.