headroomlabs-ai/headroom · error · SystemExit

Error: litellm required for --answer-model. Run: pip install

Error message

Error: litellm required for --answer-model. Run: pip install litellm

What it means

When the memory-eval command is given --answer-model, it answers questions via litellm.completion. litellm is optional; if importing it fails, the command prints 'Error: litellm required for --answer-model. Run: pip install litellm' and exits 1. Only the --answer-model code path needs it — evals without an answer model run fine.

Source

Thrown at headroom/cli/evals.py:520

3. For "when" questions: give the specific date if mentioned (e.g., "7 May 2023", "2022")
4. For "what" questions: give the specific thing/action
5. For "who" questions: give the name
6. If the exact answer is in the memories, use those exact words/dates
7. If you cannot find the answer, say "Information not found"

## Answer (be concise - just the facts):"""

                response = litellm.completion(
                    model=answer_model,
                    messages=[{"role": "user", "content": prompt}],
                    temperature=0.0,
                    max_tokens=150,
                )
                return response.choices[0].message.content or ""

        except ImportError:
            click.echo("Error: litellm required for --answer-model. Run: pip install litellm")
            raise SystemExit(1) from None

    # Create LLM judge if enabled
    llm_judge_fn: Callable[[str, str, str], tuple[float, str]] | None = None
    if llm_judge:
        # Use answer model for judge if not explicitly set
        effective_judge_model = judge_model
        if answer_model and judge_model == "gpt-4o":
            effective_judge_model = answer_model  # Match the answer model

        if judge_provider == "simple":
            llm_judge_fn = simple_judge
        elif judge_provider == "openai":
            llm_judge_fn = create_openai_judge(model=effective_judge_model)
        elif judge_provider == "anthropic":
            llm_judge_fn = create_anthropic_judge(model=effective_judge_model)
        else:
            llm_judge_fn = create_litellm_judge(model=effective_judge_model)

View on GitHub (pinned to 322425c43b)

Solutions

1. pip install litellm (or reinstall with an extra that includes it)
2. Drop --answer-model to use the default answering path that does not require litellm
3. Set your API key env vars (OPENAI_API_KEY etc.) once litellm is installed so the completion call succeeds
4. Verify: python -c "import litellm; print(litellm.__version__)"

Example fix

# before
$ headroom evals memory --answer-model gpt-4o-mini
# Error: litellm required for --answer-model.

# after
$ pip install litellm
$ headroom evals memory --answer-model gpt-4o-mini

try:
    import litellm  # noqa: F401
    HAS_LITELLM = True
except ImportError:
    HAS_LITELLM = False

if answer_model and not HAS_LITELLM:
    raise SystemExit("--answer-model requires litellm: pip install litellm")

When it happens

Trigger: Running `headroom evals memory --answer-model gpt-4o-mini ...` (or --judge-provider litellm paths) without litellm installed in the active environment.

Common situations: Installing [evals] extra but not litellm (it is a separate dependency); running evals on a slim CI image; uninstalling litellm to slim an image and forgetting this path.

Related errors

• Error: Memory eval dependencies not installed.
• Error: Memory eval V2 dependencies not installed.
• Evaluation interrupted.
• Error: MCP SDK not installed.
• Error: MCP dependencies not installed: {e}

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/3218fb77c84e478b. Report an issue: GitHub.