headroomlabs-ai/headroom · error · ValueError

unsupported headroom-bench proxy mode: {raw_mode!r}

Error message

unsupported headroom-bench proxy mode: {raw_mode!r}

What it means

Raised by HarnessScenario.bench_arms() when the proxy is enabled (proxy_config.optimize is true) but proxy_config.mode is neither 'token' nor 'cache'. The bench harness maps the proxy mode onto headroom-bench arms (A0 direct, plus optimization arms) and only token/cache are meaningful benchmark modes, so anything else is rejected.

Source

Thrown at headroom/testing/harness.py:1170

        if self.proxy_config.disable_kompress:
            cmd.append("--disable-kompress")
        if self.proxy_config.lossless:
            cmd.append("--lossless")
        if self.proxy_config.bedrock_region:
            cmd += ["--bedrock-region", self.proxy_config.bedrock_region]
        if self.proxy_config.bedrock_profile:
            cmd += ["--bedrock-profile", self.proxy_config.bedrock_profile]
        cmd += list(extra_flags)
        return tuple(cmd)

    def bench_arms(
        self, *, provider: Literal["anthropic", "openai"] = "anthropic"
    ) -> tuple[BenchArm, ...]:
        headroom_proxy_mode: Literal["off", "token", "cache"]
        if self.proxy_config.optimize:
            raw_mode = cast(str, self.proxy_config.mode)
            if raw_mode not in {"token", "cache"}:
                raise ValueError(f"unsupported headroom-bench proxy mode: {raw_mode!r}")
            headroom_proxy_mode = cast(Literal["token", "cache"], raw_mode)
        else:
            headroom_proxy_mode = "off"
        return (
            BenchArm(
                name=ArmName.A0_DIRECT,
                provider=provider,
                proxy_mode=None,
                proxy_flags=(),
                label="Direct provider API",
            ),
            BenchArm(
                name=ArmName.A1_PASSTHROUGH,
                provider=provider,
                proxy_mode="off",
                proxy_flags=(),
                label="Headroom proxy passthrough",
            ),

View on GitHub (pinned to 322425c43b)

Solutions

1. Set an explicit supported mode: configure_proxy(optimize=True, mode='token') or mode='cache'.
2. If you intended no proxy optimization, set optimize=False so bench_arms uses the 'off' arm.
3. Validate mode membership {'token','cache'} in test setup when optimize is enabled.

Example fix

# before
headroom.configure_proxy(optimize=True)  # mode unset/invalid
arms = scenario.bench_arms()

# after
headroom.configure_proxy(optimize=True, mode="cache")
arms = scenario.bench_arms()

if proxy_config.optimize:
    assert proxy_config.mode in {"token", "cache"}, (
        f"bench requires proxy mode 'token' or 'cache', got {proxy_config.mode!r}")

def bench_ready(cfg) -> bool:
    return (not cfg.optimize) or cfg.mode in {"token", "cache"}

try:
    arms = scenario.bench_arms()
except ValueError as e:
    if "unsupported headroom-bench proxy mode" in str(e):
        scenario.proxy_config.mode = "token"
        arms = scenario.bench_arms()
    else:
        raise

When it happens

Trigger: Building a scenario with configure_proxy(optimize=True) but leaving mode at its default (or setting mode='off', 'auto', 'kompress', etc.) and then calling scenario.bench_arms() or running the suite's benchmark path.

Common situations: Copying a proxy config from the main proxy server (which supports more mode strings) into a bench scenario; enabling optimize without deciding which optimization to measure; version changes that renamed proxy modes.

Related errors

• Unknown provider: {self.provider}
• OPENAI_API_KEY environment variable required
• api_key is required for cloud mode
• default_importance must be 0.0-1.0, got {self.default_import
• dedup_similarity_threshold must be 0.0-1.0, got {self.dedup_

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/1afc3fe83200b44c. Report an issue: GitHub.