headroomlabs-ai/headroom · error · RuntimeError

request was refused by safety classifiers

Error message

request was refused by safety classifiers

What it means

run() in the eval shaper script calls Anthropic's Messages API and inspects response.stop_reason. A stop_reason of `refusal` means the model declined to continue because safety classifiers flagged the request content — this is an API-level refusal, not a network or schema error, and the response contains no usable output.

Source

Thrown at scripts/eval_output_shaper.py:171

                        "tool_use_id": "toolu_eval_01",
                        "content": BUGGY_CODE,
                    }
                ],
            },
        ],
    }


def run(client: anthropic.Anthropic, body: dict[str, Any]) -> dict[str, int]:
    # The installed SDK may predate output_config as a typed kwarg; the API
    # accepts it either way, so pass it through extra_body.
    body = dict(body)
    extra_body = None
    if "output_config" in body:
        extra_body = {"output_config": body.pop("output_config")}
    response = client.messages.create(**body, extra_body=extra_body)
    if response.stop_reason == "refusal":
        raise RuntimeError("request was refused by safety classifiers")
    return {
        "input_tokens": response.usage.input_tokens,
        "output_tokens": response.usage.output_tokens,
    }


def main() -> int:
    load_env()
    if not os.environ.get("ANTHROPIC_API_KEY"):
        print("ANTHROPIC_API_KEY not found (env or .env)", file=sys.stderr)
        return 1
    client = anthropic.Anthropic()
    which = sys.argv[1].upper() if len(sys.argv) > 1 else "ALL"

    conditions: list[tuple[str, str, dict[str, Any]]] = []

    if which in ("A", "ALL"):
        # Scenario A: baseline vs steered.

View on GitHub (pinned to 322425c43b)

Solutions

1. Log the request body (minus secrets) when stop_reason == 'refusal' to identify which item tripped the classifier.
2. Rewrite or drop the offending prompt content; split sensitive examples out of the batch.
3. Retry the single item once — classifiers occasionally flag borderline content non-deterministically.
4. If a whole eval set refuses consistently, audit the shared system prompt first.

Example fix

// before
response = client.messages.create(**body, extra_body=extra_body)
if response.stop_reason == "refusal":
    raise RuntimeError("request was refused by safety classifiers")

// after: tag the failing item and continue the batch
response = client.messages.create(**body, extra_body=extra_body)
if response.stop_reason == "refusal":
    refusals.append({"item": item_id, "body": body})
    continue

def safe_to_send(text: str) -> bool:
    # cheap pre-filter for content known to trip classifiers; keep domain-specific
    blocked = ["<pattern-that-always-refuses>"]
    return not any(b in text for b in blocked)

def is_refusal(response) -> bool:
    return getattr(response, "stop_reason", None) == "refusal"

try:
    response = client.messages.create(**body, extra_body=extra_body)
except anthropic.APIStatusError as e:
    raise  # transport/HTTP errors are not refusals; handle separately
if response.stop_reason == "refusal":
    # content refusal: quarantine item, optionally retry once, continue batch
    refusals.append(item_id)
    continue

When it happens

Trigger: Prompts or few-shot examples containing content that trips Anthropic's safety filters; a system prompt asking the model to produce disallowed output; edge cases in shaper prompts (e.g., formatting instructions that read as manipulation attempts).

Common situations: Running evals over corpora that include sensitive user content; aggressive prompt templates; an empty-but-flagged input combination. Refusals are content-dependent and can appear intermittently across a batch.

Related errors

• anthropic package required. Install with: pip install anthro
• openai package required. Install with: pip install openai
• Unknown provider: {self.provider}
• anthropic package required. Install with: pip install anthro
• `{' '.join(cmd)}` returned unparseable output. First {_MAX_S

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/84a975d789299c11. Report an issue: GitHub.