headroomlabs-ai/headroom · error · ValueError

Unknown compression-only benchmark: {spec.name}

Error message

Unknown compression-only benchmark: {spec.name}

What it means

Raised by SuiteRunner when running a 'compression-only' benchmark spec whose name does not match either of the two implemented benchmarks: 'CCR Round-trip' or 'Info Retention'. The dispatch is a hard-coded if/elif on the human-readable spec name, so any renaming, casing change, or new benchmark idea in a suite definition file fails here instead of silently skipping.

Source

Thrown at headroom/evals/suite_runner.py:476

        }

    def _run_compression_only_benchmark(
        self,
        spec: BenchmarkSpec,
    ) -> dict[str, Any]:
        """Run a compression-only benchmark (zero LLM cost)."""
        from headroom.evals.runners.compression_only import CompressionOnlyRunner

        runner = CompressionOnlyRunner()

        if spec.name == "CCR Round-trip":
            cases = runner.generate_ccr_test_cases(n=spec.sample_size)
            result = runner.evaluate_ccr_lossless(cases)
        elif spec.name == "Info Retention":
            cases = runner.generate_info_retention_cases(n=spec.sample_size)
            result = runner.evaluate_information_retention(cases)
        else:
            raise ValueError(f"Unknown compression-only benchmark: {spec.name}")

        return {
            "accuracy_rate": result.accuracy_rate,
            "avg_compression_ratio": result.avg_compression_ratio,
            "tokens_saved": result.total_tokens_saved,
            "passed": result.passed,
            "n_samples": result.total_cases,
            "duration_seconds": result.duration_seconds,
        }

    def run(self) -> SuiteResult:
        """Run the full evaluation suite."""
        from headroom.evals.cost_tracker import CostTracker
        from headroom.evals.reports.report_card import BenchmarkRunResult, SuiteResult

        specs = self._get_specs()
        tracker = CostTracker(budget_usd=self.budget_usd)
        results: list[BenchmarkRunResult] = []

View on GitHub (pinned to 322425c43b)

Solutions

1. Use the exact names: spec.name = 'CCR Round-trip' or 'Info Retention' (mind capitalization and the hyphen/space).
2. Copy a known-good bundled suite file and edit sample_size instead of authoring names from scratch.
3. If you intended a new benchmark, implement a branch in run_compression_only / CompressionOnlyRunner first.
4. Validate suite files against the shipped example suite before a long run.

Example fix

# before
{"name": "ccr round-trip", "type": "compression-only", "sample_size": 50}

# after
{"name": "CCR Round-trip", "type": "compression-only", "sample_size": 50}

COMPRESSION_ONLY_BENCHMARKS = {"CCR Round-trip", "Info Retention"}

for spec in suite.compression_only_specs:
    if spec.name not in COMPRESSION_ONLY_BENCHMARKS:
        raise SystemExit(
            f"unknown compression-only benchmark {spec.name!r}; "
            f"valid: {sorted(COMPRESSION_ONLY_BENCHMARKS)}"
        )

def is_known_compression_benchmark(name: str) -> bool:
    return name in {"CCR Round-trip", "Info Retention"}

try:
    suite_runner.run()
except ValueError as e:
    if str(e).startswith("Unknown compression-only benchmark"):
        sys.exit(f"fix suite file: {e}")
    raise

When it happens

Trigger: A suite JSON/YAML containing a CompressionOnly spec named e.g. 'ccr round-trip', 'CCR Roundtrip', 'Round-trip', or a not-yet-implemented benchmark like 'Token Overhead'. The name must match the literal strings in suite_runner.py exactly.

Common situations: Hand-writing suite definitions without copying the canonical names; renaming benchmarks in docs but not code (or vice versa); suite files shared across headroom versions where names drifted.

Related errors

• Unknown provider: {self.llm_config.provider}
• position must be one of {POSITIONS}, got {position!r}
• Unknown provider: {self.provider}
• invalid pipeline config TOML: {0}
• recommendations file not found: {0}

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/2824c72f0a03aeee. Report an issue: GitHub.