headroomlabs-ai/headroom · error · ImportError

LiteLLM package required for LLM judge. Install with: pip in

Error message

LiteLLM package required for LLM judge. Install with: pip install litellm

What it means

Error "LiteLLM package required for LLM judge. Install with: pip install litellm" thrown in headroomlabs-ai/headroom.

Source

Thrown at headroom/evals/memory/judge.py:170

) -> Callable[[str, str, str], tuple[float, str]]:
    """Create an LLM judge using LiteLLM for any supported provider.

    Args:
        model: Model identifier (e.g., "gpt-4o", "claude-sonnet-4-20250514", "ollama/llama3").

    Returns:
        A judge function that takes (question, ground_truth, prediction)
        and returns (score, reasoning).

    Example:
        # Use Ollama for local evaluation
        judge_fn = create_litellm_judge(model="ollama/llama3")
        score, reasoning = judge_fn(question, ground_truth, prediction)
    """
    try:
        import litellm
    except ImportError as e:
        raise ImportError(
            "LiteLLM package required for LLM judge. Install with: pip install litellm"
        ) from e

    def judge(question: str, ground_truth: str, prediction: str) -> tuple[float, str]:
        prompt = JUDGE_PROMPT.format(
            question=question,
            ground_truth=ground_truth,
            prediction=prediction,
        )

        response = litellm.completion(
            model=model,
            messages=[{"role": "user", "content": prompt}],
            temperature=0.0,
            max_tokens=200,
        )

        text = response.choices[0].message.content or ""

View on GitHub (pinned to 322425c43b)

Solutions

1. Install LiteLLM: pip install litellm
2. Or install the evals extra: pip install headroom-ai[evals]

When it happens

Trigger: Raised when the memory eval LLM judge routes through LiteLLM but the `litellm` package is not installed.

Common situations: See trigger scenarios.

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/fe5e2f922dd0d09e. Report an issue: GitHub.