{"record":{"id":"84a975d789299c11","repo":"headroomlabs-ai/headroom","slug":"request-was-refused-by-safety-classifiers","errorCode":null,"errorMessage":"request was refused by safety classifiers","messagePattern":"request was refused by safety classifiers","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"scripts/eval_output_shaper.py","lineNumber":171,"sourceCode":"                        \"tool_use_id\": \"toolu_eval_01\",\n                        \"content\": BUGGY_CODE,\n                    }\n                ],\n            },\n        ],\n    }\n\n\ndef run(client: anthropic.Anthropic, body: dict[str, Any]) -> dict[str, int]:\n    # The installed SDK may predate output_config as a typed kwarg; the API\n    # accepts it either way, so pass it through extra_body.\n    body = dict(body)\n    extra_body = None\n    if \"output_config\" in body:\n        extra_body = {\"output_config\": body.pop(\"output_config\")}\n    response = client.messages.create(**body, extra_body=extra_body)\n    if response.stop_reason == \"refusal\":\n        raise RuntimeError(\"request was refused by safety classifiers\")\n    return {\n        \"input_tokens\": response.usage.input_tokens,\n        \"output_tokens\": response.usage.output_tokens,\n    }\n\n\ndef main() -> int:\n    load_env()\n    if not os.environ.get(\"ANTHROPIC_API_KEY\"):\n        print(\"ANTHROPIC_API_KEY not found (env or .env)\", file=sys.stderr)\n        return 1\n    client = anthropic.Anthropic()\n    which = sys.argv[1].upper() if len(sys.argv) > 1 else \"ALL\"\n\n    conditions: list[tuple[str, str, dict[str, Any]]] = []\n\n    if which in (\"A\", \"ALL\"):\n        # Scenario A: baseline vs steered.","sourceCodeStart":153,"sourceCodeEnd":189,"githubUrl":"https://github.com/headroomlabs-ai/headroom/blob/322425c43bffde1ed0b64fecf3cf5951565dd82b/scripts/eval_output_shaper.py#L153-L189","documentation":"run() in the eval shaper script calls Anthropic's Messages API and inspects response.stop_reason. A stop_reason of `refusal` means the model declined to continue because safety classifiers flagged the request content — this is an API-level refusal, not a network or schema error, and the response contains no usable output.","triggerScenarios":"Prompts or few-shot examples containing content that trips Anthropic's safety filters; a system prompt asking the model to produce disallowed output; edge cases in shaper prompts (e.g., formatting instructions that read as manipulation attempts).","commonSituations":"Running evals over corpora that include sensitive user content; aggressive prompt templates; an empty-but-flagged input combination. Refusals are content-dependent and can appear intermittently across a batch.","solutions":["Log the request body (minus secrets) when stop_reason == 'refusal' to identify which item tripped the classifier.","Rewrite or drop the offending prompt content; split sensitive examples out of the batch.","Retry the single item once — classifiers occasionally flag borderline content non-deterministically.","If a whole eval set refuses consistently, audit the shared system prompt first."],"exampleFix":"// before\nresponse = client.messages.create(**body, extra_body=extra_body)\nif response.stop_reason == \"refusal\":\n    raise RuntimeError(\"request was refused by safety classifiers\")\n\n// after: tag the failing item and continue the batch\nresponse = client.messages.create(**body, extra_body=extra_body)\nif response.stop_reason == \"refusal\":\n    refusals.append({\"item\": item_id, \"body\": body})\n    continue","handlingStrategy":"retry","validationCode":"def safe_to_send(text: str) -> bool:\n    # cheap pre-filter for content known to trip classifiers; keep domain-specific\n    blocked = [\"<pattern-that-always-refuses>\"]\n    return not any(b in text for b in blocked)","typeGuard":"def is_refusal(response) -> bool:\n    return getattr(response, \"stop_reason\", None) == \"refusal\"","tryCatchPattern":"try:\n    response = client.messages.create(**body, extra_body=extra_body)\nexcept anthropic.APIStatusError as e:\n    raise  # transport/HTTP errors are not refusals; handle separately\nif response.stop_reason == \"refusal\":\n    # content refusal: quarantine item, optionally retry once, continue batch\n    refusals.append(item_id)\n    continue","preventionTips":["Run eval batches with per-item error handling so one refusal does not kill the run.","Keep prompt templates neutral; avoid instructions that pattern-match manipulation tactics.","Log item IDs alongside refusals to build a blocklist of tripping content."],"tags":["anthropic","llm","api","safety","eval"],"backgroundTag":null,"analyzedSha":"322425c43bffde1ed0b64fecf3cf5951565dd82b","analyzedAt":"2026-08-15T01:03:05.481Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}