headroomlabs-ai/headroom · error · SystemExit

Error: Memory eval dependencies not installed.

Error message

Error: Memory eval dependencies not installed.

What it means

The 'headroom evals memory' CLI command imports its evaluator stack (LoCoMoEvaluator, MemoryEvalConfig, judge factories, headroom.memory) inside the command. If any of these optional memory/evals dependencies is missing, the ImportError is caught, a message pointing at 'pip install headroom[memory,evals]' is printed, and the process exits with code 1 via SystemExit. It is a dependency-guard exit, not a data error.

Source

Thrown at headroom/cli/evals.py:454

    warnings.filterwarnings("ignore", message=".*Pydantic serializer warnings.*")
    warnings.filterwarnings("ignore", category=UserWarning, module="pydantic")

    try:
        from headroom.evals.memory import (
            LoCoMoEvaluator,
            MemoryEvalConfig,
            create_anthropic_judge,
            create_litellm_judge,
            create_openai_judge,
            simple_judge,
        )
        from headroom.memory import MemoryConfig
    except ImportError as e:
        click.echo("Error: Memory eval dependencies not installed.")
        click.echo("Run: pip install headroom[memory,evals]")
        click.echo(f"Details: {e}")
        raise SystemExit(1) from None

    import asyncio

    # Build configuration
    parsed_categories = _parse_categories(categories)

    memory_config = MemoryConfig()

    eval_config = MemoryEvalConfig(
        n_conversations=n_conversations,
        categories=parsed_categories,
        skip_adversarial=not include_adversarial,
        top_k_memories=top_k,
        llm_judge_enabled=llm_judge,
        llm_judge_model=judge_model,
        memory_config=memory_config,
        f1_threshold=f1_threshold,
        extract_memories=not no_extract,

View on GitHub (pinned to 322425c43b)

Solutions

1. Install both extras: pip install 'headroom-ai[memory,evals]'
2. Read the 'Details:' line to see exactly which module failed and install it directly if the extra name differs across versions
3. Confirm you are in the same virtualenv headroom runs from (which headroom; which python)
4. Upgrade headroom so the extras metadata matches the imports the CLI performs

Example fix

# before
$ headroom evals memory --n-conversations 2
# Error: Memory eval dependencies not installed.

# after
$ pip install 'headroom-ai[memory,evals]'
$ headroom evals memory --n-conversations 2

import subprocess, sys
missing = subprocess.run(
    [sys.executable, "-c", "import headroom.evals.memory"],
    capture_output=True,
).returncode != 0
if missing:
    raise SystemExit("install extras first: pip install 'headroom-ai[memory,evals]'")

proc = subprocess.run(["headroom", "evals", "memory", ...], capture_output=True, text=True)
if proc.returncode != 0 and "not installed" in proc.stderr + proc.stdout:
    print("eval extras missing in runner env — install and retry")

When it happens

Trigger: Running `headroom evals memory ...` in an environment without the [memory,evals] extras (e.g. missing lancedb, eval datasets, or judge SDK deps); the failing import name is shown in 'Details: {e}'.

Common situations: Bare pip install headroom-ai; CI images trimmed of optional extras; Python version where an eval dependency has no wheel; mixing pip and system site-packages so the extra resolves in a different interpreter.

Related errors

• Error: Memory eval V2 dependencies not installed.
• Error: litellm required for --answer-model. Run: pip install
• Error: MCP SDK not installed.
• Error: MCP dependencies not installed: {e}
• Error: Proxy dependencies not installed. Run: pip install he

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/8bc6ce0e47586e60. Report an issue: GitHub.