{"record":{"id":"2824c72f0a03aeee","repo":"headroomlabs-ai/headroom","slug":"unknown-compression-only-benchmark-spec-name","errorCode":null,"errorMessage":"Unknown compression-only benchmark: {spec.name}","messagePattern":"Unknown compression-only benchmark: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"headroom/evals/suite_runner.py","lineNumber":476,"sourceCode":"        }\n\n    def _run_compression_only_benchmark(\n        self,\n        spec: BenchmarkSpec,\n    ) -> dict[str, Any]:\n        \"\"\"Run a compression-only benchmark (zero LLM cost).\"\"\"\n        from headroom.evals.runners.compression_only import CompressionOnlyRunner\n\n        runner = CompressionOnlyRunner()\n\n        if spec.name == \"CCR Round-trip\":\n            cases = runner.generate_ccr_test_cases(n=spec.sample_size)\n            result = runner.evaluate_ccr_lossless(cases)\n        elif spec.name == \"Info Retention\":\n            cases = runner.generate_info_retention_cases(n=spec.sample_size)\n            result = runner.evaluate_information_retention(cases)\n        else:\n            raise ValueError(f\"Unknown compression-only benchmark: {spec.name}\")\n\n        return {\n            \"accuracy_rate\": result.accuracy_rate,\n            \"avg_compression_ratio\": result.avg_compression_ratio,\n            \"tokens_saved\": result.total_tokens_saved,\n            \"passed\": result.passed,\n            \"n_samples\": result.total_cases,\n            \"duration_seconds\": result.duration_seconds,\n        }\n\n    def run(self) -> SuiteResult:\n        \"\"\"Run the full evaluation suite.\"\"\"\n        from headroom.evals.cost_tracker import CostTracker\n        from headroom.evals.reports.report_card import BenchmarkRunResult, SuiteResult\n\n        specs = self._get_specs()\n        tracker = CostTracker(budget_usd=self.budget_usd)\n        results: list[BenchmarkRunResult] = []","sourceCodeStart":458,"sourceCodeEnd":494,"githubUrl":"https://github.com/headroomlabs-ai/headroom/blob/322425c43bffde1ed0b64fecf3cf5951565dd82b/headroom/evals/suite_runner.py#L458-L494","documentation":"Raised by SuiteRunner when running a 'compression-only' benchmark spec whose name does not match either of the two implemented benchmarks: 'CCR Round-trip' or 'Info Retention'. The dispatch is a hard-coded if/elif on the human-readable spec name, so any renaming, casing change, or new benchmark idea in a suite definition file fails here instead of silently skipping.","triggerScenarios":"A suite JSON/YAML containing a CompressionOnly spec named e.g. 'ccr round-trip', 'CCR Roundtrip', 'Round-trip', or a not-yet-implemented benchmark like 'Token Overhead'. The name must match the literal strings in suite_runner.py exactly.","commonSituations":"Hand-writing suite definitions without copying the canonical names; renaming benchmarks in docs but not code (or vice versa); suite files shared across headroom versions where names drifted.","solutions":["Use the exact names: spec.name = 'CCR Round-trip' or 'Info Retention' (mind capitalization and the hyphen/space).","Copy a known-good bundled suite file and edit sample_size instead of authoring names from scratch.","If you intended a new benchmark, implement a branch in run_compression_only / CompressionOnlyRunner first.","Validate suite files against the shipped example suite before a long run."],"exampleFix":"# before\n{\"name\": \"ccr round-trip\", \"type\": \"compression-only\", \"sample_size\": 50}\n\n# after\n{\"name\": \"CCR Round-trip\", \"type\": \"compression-only\", \"sample_size\": 50}","handlingStrategy":"validation","validationCode":"COMPRESSION_ONLY_BENCHMARKS = {\"CCR Round-trip\", \"Info Retention\"}\n\nfor spec in suite.compression_only_specs:\n    if spec.name not in COMPRESSION_ONLY_BENCHMARKS:\n        raise SystemExit(\n            f\"unknown compression-only benchmark {spec.name!r}; \"\n            f\"valid: {sorted(COMPRESSION_ONLY_BENCHMARKS)}\"\n        )","typeGuard":"def is_known_compression_benchmark(name: str) -> bool:\n    return name in {\"CCR Round-trip\", \"Info Retention\"}","tryCatchPattern":"try:\n    suite_runner.run()\nexcept ValueError as e:\n    if str(e).startswith(\"Unknown compression-only benchmark\"):\n        sys.exit(f\"fix suite file: {e}\")\n    raise","preventionTips":["Start suite definitions from the shipped example file rather than typing names.","Add a suite-lint step that checks all spec names against the known set in CI.","Treat benchmark names as APIs: check for renames on every headroom upgrade."],"tags":["validation","suite-config","evals","naming","typo"],"backgroundTag":null,"analyzedSha":"322425c43bffde1ed0b64fecf3cf5951565dd82b","analyzedAt":"2026-08-15T01:03:05.481Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}