headroomlabs-ai/headroom · error · ImportError

OpenAI package required for LLM judge. Install with: pip ins

Error message

OpenAI package required for LLM judge. Install with: pip install openai

What it means

Error "OpenAI package required for LLM judge. Install with: pip install openai" thrown in headroomlabs-ai/headroom.

Source

Thrown at headroom/evals/memory/judge.py:72

        model: OpenAI model to use (default: gpt-4o).
        api_key: OpenAI API key (uses OPENAI_API_KEY env var if not provided).

    Returns:
        A judge function that takes (question, ground_truth, prediction)
        and returns (score, reasoning).

    Example:
        judge_fn = create_openai_judge(model="gpt-4o-mini")
        score, reasoning = judge_fn(
            "What is Alice's favorite color?",
            "Blue",
            "Alice prefers blue"
        )
    """
    try:
        from openai import OpenAI
    except ImportError as e:
        raise ImportError(
            "OpenAI package required for LLM judge. Install with: pip install openai"
        ) from e

    client = OpenAI(api_key=api_key) if api_key else OpenAI()

    def judge(question: str, ground_truth: str, prediction: str) -> tuple[float, str]:
        prompt = JUDGE_PROMPT.format(
            question=question,
            ground_truth=ground_truth,
            prediction=prediction,
        )

        response = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": prompt}],
            temperature=0.0,  # Deterministic scoring
            max_tokens=200,
        )

View on GitHub (pinned to 322425c43b)

Solutions

1. Install the OpenAI package: pip install openai
2. Set OPENAI_API_KEY in the environment before running the judge
3. Or install the evals extra: pip install headroom-ai[evals]

When it happens

Trigger: Raised when the memory eval LLM judge is configured with the OpenAI provider but the `openai` package is not installed.

Common situations: See trigger scenarios.

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/f4f6e8e141ba51b5. Report an issue: GitHub.