{"record":{"id":"998cf102ff64f5a6","repo":"headroomlabs-ai/headroom","slug":"headroom-modified-prompt-semantics-differences","errorCode":null,"errorMessage":"Headroom modified prompt semantics! Differences: {result.differences}. Reasoning: {result.reasoning}","messagePattern":"Headroom modified prompt semantics! Differences: (.+?)\\. Reasoning: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"headroom/evals/prompt_comparison.py","lineNumber":630,"sourceCode":"\n    Example:\n        # Capture messages before and after Headroom\n        original = [{\"role\": \"user\", \"content\": \"Hello, how are you?\"}]\n        after_headroom = [{\"role\": \"user\", \"content\": \"Hello, how are you?\"}]\n\n        result = verify_headroom_preservation(\n            original, after_headroom, fail_on_difference=True\n        )\n    \"\"\"\n    result = compare_messages(\n        original_messages=original_messages,\n        modified_messages=headroom_messages,\n        judge_model=judge_model,\n        api_key=api_key,\n    )\n\n    if fail_on_difference and not result.are_equivalent:\n        raise ValueError(\n            f\"Headroom modified prompt semantics! \"\n            f\"Differences: {result.differences}. \"\n            f\"Reasoning: {result.reasoning}\"\n        )\n\n    return result\n","sourceCodeStart":612,"sourceCodeEnd":637,"githubUrl":"https://github.com/headroomlabs-ai/headroom/blob/322425c43bffde1ed0b64fecf3cf5951565dd82b/headroom/evals/prompt_comparison.py#L612-L637","documentation":"Raised by verify_headroom_preservation when the LLM judge decided the Headroom-transformed prompt is NOT semantically equivalent to the original and fail_on_difference=True (the default for this verification helper). This is a correctness alarm, not an infrastructure failure: the judge prompt response reported concrete differences and reasoning, both embedded in the message. It signals a possible destructive bug in the compression/context-reduction pipeline.","triggerScenarios":"Calling verify_headroom_preservation(original, after_headroom, fail_on_difference=True) where the judge model scores the messages as non-equivalent — e.g. compression dropped a constraint, tool definition, or instruction; round-tripping through CCR lost content; or the judge model is weak/noisy and hallucinates a difference.","commonSituations":"CI gating prompt-transform changes; after upgrading Headroom or editing a transform; using a cheap judge model that produces false positives; feeding prompts that legitimately change semantics (system-prompt injection tests).","solutions":["Inspect result.differences and result.reasoning from a non-failing run (fail_on_difference=False) to see exactly what the judge thinks changed.","If the judge is wrong, pass a stronger judge_model (e.g. a frontier model) and re-run — noisy judges are the most common false positive.","If the difference is real, bisect the transform pipeline (disable transforms one at a time) to find which stage mutates semantics, and fix or exclude it for these messages.","Only as a deliberate policy, call with fail_on_difference=False to log-and-continue instead of raising."],"exampleFix":"# before\nresult = verify_headroom_preservation(original, after_headroom, fail_on_difference=True)\n\n# after: triage first, then decide\nresult = verify_headroom_preservation(original, after_headroom, fail_on_difference=False)\nif not result.are_equivalent:\n    logger.warning(\"judge diffs=%s reasoning=%s\", result.differences, result.reasoning)\n    # gate only on confirmed regressions\n","handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"from headroom.evals.prompt_comparison import verify_headroom_preservation\ntry:\n    verify_headroom_preservation(original, after_headroom, fail_on_difference=True)\nexcept ValueError as e:\n    if \"modified prompt semantics\" in str(e):\n        # triage, don't crash the whole suite\n        logger.error(\"semantic regression: %s\", e)\n        record_failure_and_continue()\n    else:\n        raise","preventionTips":["Run verify with fail_on_difference=False in exploratory mode; reserve fail_on_difference=True for release gates.","Use a strong, deterministic judge model (temperature 0 if configurable) to cut false positives.","Keep a corpus of known-good (original, headroom) pairs as regression fixtures for the transform pipeline."],"tags":["semantics","llm-judge","evals","prompt-preservation","regression"],"backgroundTag":null,"analyzedSha":"322425c43bffde1ed0b64fecf3cf5951565dd82b","analyzedAt":"2026-08-15T01:03:05.481Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}