{"record":{"id":"4e983872451d4c9a","repo":"headroomlabs-ai/headroom","slug":"headroomsuite-requires-at-least-one-scenario","errorCode":null,"errorMessage":"HeadroomSuite requires at least one scenario","messagePattern":"HeadroomSuite requires at least one scenario","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"headroom/testing/harness.py","lineNumber":1094,"sourceCode":"            benchmark_ref=benchmark_ref,\n            provider=provider,\n            now=now,\n        )\n        written: list[Path] = []\n        for scenario_name, manifest in manifests.items():\n            target = target_dir / f\"{scenario_name}.agent-evals.json\"\n            target.write_text(\n                json.dumps(manifest.to_dict(), indent=2, sort_keys=True) + \"\\n\",\n                encoding=\"utf-8\",\n            )\n            written.append(target)\n        return tuple(written)\n\n    WriteAgentEvalsManifests = write_agent_evals_manifests\n\n    def _require_scenarios(self) -> tuple[HarnessScenario, ...]:\n        if not self._scenarios:\n            raise ValueError(\"HeadroomSuite requires at least one scenario\")\n        return tuple(self._scenarios)\n\n\n@dataclass(frozen=True)\nclass HarnessScenario:\n    \"\"\"A fully-built, immutable Headroom test scenario.\"\"\"\n\n    name: str\n    provider: ProviderTarget\n    platform: PlatformTarget\n    headroom_config: HeadroomConfig\n    proxy_config: ProxyConfig\n    metadata: dict[str, Any]\n\n    @property\n    def contract(self) -> ScenarioContract:\n        return ScenarioContract(\n            headroom_fields=_field_contract(\"headroom\", HeadroomConfig),","sourceCodeStart":1076,"sourceCodeEnd":1112,"githubUrl":"https://github.com/headroomlabs-ai/headroom/blob/322425c43bffde1ed0b64fecf3cf5951565dd82b/headroom/testing/harness.py#L1076-L1112","documentation":"Raised by HeadroomSuite._require_scenarios(), a guard invoked by suite operations (run, manifest writing, etc.) when the suite's internal scenario list is empty. The suite cannot do anything meaningful with zero scenarios, so it fails fast with a clear message instead of producing empty reports.","triggerScenarios":"Creating HeadroomSuite() and immediately calling run(), write_agent_evals_manifests(...), or any operation that routes through _require_scenarios() without ever calling add()/extend(); a filter step that removes all scenarios before running.","commonSituations":"Suite assembled conditionally from a config-driven list that ends up empty (bad filter, wrong env var, empty glob); CI parameter sets that select zero scenarios; refactoring that moved add() calls behind a flag that is off.","solutions":["Add at least one scenario before running: `suite.add(HarnessScenario(...))` or `suite.add(headroom_builder)`.","If scenarios come from a dynamic list, assert or warn when it is empty before constructing the suite.","Check the filter/config that produces the scenario list (env var, file glob, model allowlist)."],"exampleFix":"# before\nsuite = HeadroomSuite()\nreport = suite.run()  # ValueError\n\n# after\nsuite = HeadroomSuite().add(build_baseline())\nreport = suite.run()","handlingStrategy":"validation","validationCode":"if not suite.scenarios:\n    raise RuntimeError(\"no scenarios configured; check suite setup\")\nsuite.run()","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Assert non-empty scenario lists in test/setup code.","Log the scenario count before running suites.","Validate config-driven scenario selection before constructing the suite."],"tags":["testing","harness","validation","empty-input"],"backgroundTag":null,"analyzedSha":"322425c43bffde1ed0b64fecf3cf5951565dd82b","analyzedAt":"2026-08-15T01:03:05.481Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}