{"record":{"id":"81b25fcf77d92b46","repo":"headroomlabs-ai/headroom","slug":"duplicate-scenario-name-in-suite-built-name","errorCode":null,"errorMessage":"duplicate scenario name in suite: {built.name}","messagePattern":"duplicate scenario name in suite: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"headroom/testing/harness.py","lineNumber":941,"sourceCode":"    Build = build\n    Suite = suite\n\n\nclass HeadroomSuite:\n    \"\"\"Declarative matrix of built scenarios for bench and local no-key runs.\"\"\"\n\n    def __init__(self, *, name: str = \"headroom-suite\") -> None:\n        self.name = name\n        self._scenarios: list[HarnessScenario] = []\n\n    @property\n    def scenarios(self) -> tuple[HarnessScenario, ...]:\n        return tuple(self._scenarios)\n\n    def add(self, scenario: HarnessScenario | Headroom) -> HeadroomSuite:\n        built = scenario.Build() if isinstance(scenario, Headroom) else scenario\n        if any(existing.name == built.name for existing in self._scenarios):\n            raise ValueError(f\"duplicate scenario name in suite: {built.name}\")\n        self._scenarios.append(built)\n        return self\n\n    Add = add\n\n    def extend(self, scenarios: Sequence[HarnessScenario | Headroom]) -> HeadroomSuite:\n        for scenario in scenarios:\n            self.add(scenario)\n        return self\n\n    Extend = extend\n\n    def orchestrate(\n        self,\n        tasks: Sequence[ScenarioTask],\n        *,\n        guarantees: Sequence[Guarantee] = DEFAULT_GUARANTEES,\n    ) -> ScenarioRunReport:","sourceCodeStart":923,"sourceCodeEnd":959,"githubUrl":"https://github.com/headroomlabs-ai/headroom/blob/322425c43bffde1ed0b64fecf3cf5951565dd82b/headroom/testing/harness.py#L923-L959","documentation":"Raised by HeadroomSuite.add() when a HarnessScenario being added has the same name as one already registered in the suite. Scenario names are used as unique keys (e.g. for agent-evals manifest filenames like '{name}.agent-evals.json'), so duplicates are rejected to keep outputs addressable.","triggerScenarios":"Calling suite.add(headroom) twice on builders whose .Build() produce the same default scenario name; calling suite.extend([...]) where two Headroom instances were named identically (e.g. both left as the default 'name'); re-adding a scenario after a retry loop.","commonSituations":"Building benchmark suites in a loop (names derived from a model list with duplicate entries); copy-pasting scenario builder blocks and forgetting to change .name; suites assembled from config files where two entries share a name key.","solutions":["Give each scenario a distinct name before adding: `headroom.configure_scenario(name='anthropic-cached')` or whatever naming API the builder exposes.","Check existing names first: `if s.name in {e.name for e in suite.scenarios}: rename or skip`.","Dedupe the input list (e.g. by model or config key) before suite.extend()."],"exampleFix":"# before\nsuite.add(h1).add(h2)  # both default-named\n\n# after\nh1.configure_scenario(name=\"anthropic-200k\")\nh2.configure_scenario(name=\"anthropic-1m\")\nsuite.add(h1).add(h2)","handlingStrategy":"validation","validationCode":"existing = {s.name for s in suite.scenarios}\nif built.name in existing:\n    built = built with unique name or raise/skip","typeGuard":null,"tryCatchPattern":"try:\n    suite.add(scenario)\nexcept ValueError as e:\n    if \"duplicate scenario name\" in str(e):\n        scenario.name = f\"{scenario.name}-{len(suite.scenarios)}\"\n        suite.add(scenario)\n    else:\n        raise","preventionTips":["Derive scenario names from a unique key (model+config) when building suites in loops.","Check suite.scenarios names before add().","Lint config-driven scenario lists for duplicate name keys."],"tags":["testing","harness","duplicates","config"],"backgroundTag":null,"analyzedSha":"322425c43bffde1ed0b64fecf3cf5951565dd82b","analyzedAt":"2026-08-15T01:03:05.481Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}