headroomlabs-ai/headroom · error · ValueError

Headroom modified prompt semantics! Differences: {result.dif

Error message

Headroom modified prompt semantics! Differences: {result.differences}. Reasoning: {result.reasoning}

What it means

Raised by verify_headroom_preservation when the LLM judge decided the Headroom-transformed prompt is NOT semantically equivalent to the original and fail_on_difference=True (the default for this verification helper). This is a correctness alarm, not an infrastructure failure: the judge prompt response reported concrete differences and reasoning, both embedded in the message. It signals a possible destructive bug in the compression/context-reduction pipeline.

Source

Thrown at headroom/evals/prompt_comparison.py:630

    Example:
        # Capture messages before and after Headroom
        original = [{"role": "user", "content": "Hello, how are you?"}]
        after_headroom = [{"role": "user", "content": "Hello, how are you?"}]

        result = verify_headroom_preservation(
            original, after_headroom, fail_on_difference=True
        )
    """
    result = compare_messages(
        original_messages=original_messages,
        modified_messages=headroom_messages,
        judge_model=judge_model,
        api_key=api_key,
    )

    if fail_on_difference and not result.are_equivalent:
        raise ValueError(
            f"Headroom modified prompt semantics! "
            f"Differences: {result.differences}. "
            f"Reasoning: {result.reasoning}"
        )

    return result

View on GitHub (pinned to 322425c43b)

Solutions

1. Inspect result.differences and result.reasoning from a non-failing run (fail_on_difference=False) to see exactly what the judge thinks changed.
2. If the judge is wrong, pass a stronger judge_model (e.g. a frontier model) and re-run — noisy judges are the most common false positive.
3. If the difference is real, bisect the transform pipeline (disable transforms one at a time) to find which stage mutates semantics, and fix or exclude it for these messages.
4. Only as a deliberate policy, call with fail_on_difference=False to log-and-continue instead of raising.

Example fix

# before
result = verify_headroom_preservation(original, after_headroom, fail_on_difference=True)

# after: triage first, then decide
result = verify_headroom_preservation(original, after_headroom, fail_on_difference=False)
if not result.are_equivalent:
    logger.warning("judge diffs=%s reasoning=%s", result.differences, result.reasoning)
    # gate only on confirmed regressions

from headroom.evals.prompt_comparison import verify_headroom_preservation
try:
    verify_headroom_preservation(original, after_headroom, fail_on_difference=True)
except ValueError as e:
    if "modified prompt semantics" in str(e):
        # triage, don't crash the whole suite
        logger.error("semantic regression: %s", e)
        record_failure_and_continue()
    else:
        raise

When it happens

Trigger: Calling verify_headroom_preservation(original, after_headroom, fail_on_difference=True) where the judge model scores the messages as non-equivalent — e.g. compression dropped a constraint, tool definition, or instruction; round-tripping through CCR lost content; or the judge model is weak/noisy and hallucinates a difference.

Common situations: CI gating prompt-transform changes; after upgrading Headroom or editing a transform; using a cheap judge model that produces false positives; feeding prompts that legitimately change semantics (system-prompt injection tests).

Related errors

• OpenAI API key required. Set OPENAI_API_KEY environment vari
• Error: Memory eval dependencies not installed.
• Error: litellm required for --answer-model. Run: pip install
• Evaluation interrupted.
• Error: Memory eval V2 dependencies not installed.

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/998cf102ff64f5a6. Report an issue: GitHub.