headroomlabs-ai/headroom · error · ImportError

LiteLLM package required. Install with: pip install litellm

Error message

LiteLLM package required. Install with: pip install litellm

What it means

Error "LiteLLM package required. Install with: pip install litellm" thrown in headroomlabs-ai/headroom.

Source

Thrown at headroom/evals/html_extraction.py:334

                import litellm

                def judge(question: str, ground_truth: str, prediction: str) -> tuple[float, str]:
                    prompt = HTML_JUDGE_PROMPT.format(
                        question=question,
                        ground_truth=ground_truth,
                        prediction=prediction,
                    )
                    response = litellm.completion(
                        model=self.judge_model,
                        messages=[{"role": "user", "content": prompt}],
                        temperature=0.0,
                        max_tokens=200,
                    )
                    return self._parse_judge_response(response.choices[0].message.content or "")

                return judge
            except ImportError:
                raise ImportError(
                    "LiteLLM package required. Install with: pip install litellm"
                ) from None

    def _parse_judge_response(self, text: str) -> tuple[float, str]:
        """Parse judge response to extract score and reasoning."""
        import re

        reasoning = ""
        score = 3.0  # Default

        for line in text.strip().split("\n"):
            line = line.strip()
            if line.lower().startswith("reasoning:"):
                reasoning = line[len("reasoning:") :].strip()
            elif line.lower().startswith("score:"):
                match = re.search(r"(\d+(?:\.\d+)?)", line)
                if match:
                    score = max(1.0, min(5.0, float(match.group(1))))

View on GitHub (pinned to 322425c43b)

Solutions

1. Install LiteLLM: pip install litellm
2. Or install the evals extra: pip install headroom-ai[evals]

When it happens

Trigger: Raised when the HTML extraction eval is configured to route through LiteLLM but the `litellm` package is not installed.

Common situations: See trigger scenarios.

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/65fe5e106034c1e8. Report an issue: GitHub.