headroomlabs-ai/headroom · error · RuntimeError

lm-eval failed: {result.stderr}

Error message

lm-eval failed: {result.stderr}

What it means

Raised after run_lm_eval() shells out to the lm-eval CLI (subprocess.run with HF_ALLOW_CODEEval=1 for humaneval/mbpp): a non-zero return code triggers a logged error plus RuntimeError carrying the child process's stderr. The message 'lm-eval failed' therefore wraps arbitrary downstream failures — model connection errors, missing tasks, CUDA issues, bad model_args — and stderr is the real diagnostic.

Source

Thrown at headroom/evals/comprehensive_benchmark.py:286

    # Run lm-eval
    start_time = time.time()
    env = {
        **os.environ,
        "TOKENIZERS_PARALLELISM": "false",
        "HF_ALLOW_CODE_EVAL": "1",  # Required for humaneval/mbpp tasks
    }
    result = run(
        cmd,
        capture_output=True,
        text=True,
        env=env,
    )
    duration = time.time() - start_time

    if result.returncode != 0:
        logger.error(f"lm-eval failed: {result.stderr}")
        raise RuntimeError(f"lm-eval failed: {result.stderr}")

    # Load results - lm-eval creates a directory structure with timestamped files
    results_dir: Path = Path(output_path) if output_path else Path(".")

    # Find results_*.json in the output directory (lm-eval uses timestamped filenames)
    results_file: Path | None = None
    if results_dir.is_dir():
        # Look for results_*.json files
        for f in sorted(results_dir.glob("**/results_*.json"), reverse=True):
            results_file = f
            break

    if results_file is None or not results_file.exists():
        # Parse results from stdout as fallback
        logger.warning("No results file found, parsing from stdout")
        return {"results": {}, "_duration_seconds": duration, "_stdout": result.stdout}

    with open(results_file) as fp:

View on GitHub (pinned to 322425c43b)

Solutions

1. Read the embedded stderr first — it names the actual failing component; the headroom-level message adds no extra information
2. Verify the prerequisites the subprocess assumes: lm-eval installed (`lm_eval --version`), model endpoint reachable at base_url, tasks spelled per the lm-eval task registry
3. Reproduce manually with the same command and env (HF_ALLOW_CODE_EVAL=1) to iterate faster than through headroom
4. For code tasks, confirm humaneval/mbpp execution flags are present since this runner sets them specifically for that purpose

import shutil

if shutil.which("lm_eval") is None:
    raise SystemExit("lm-eval CLI not installed — pip install lm-eval before benchmarking")

try:
    raw = run_lm_eval(model=..., tasks=tasks, base_url=base_url, ...)
except RuntimeError as e:
    raise SystemExit(f"lm-eval subprocess failed — see stderr: {e}") from e

When it happens

Trigger: run_lm_eval(model=..., model_args=..., tasks=..., base_url=...) where the lm-eval CLI exits non-zero: unreachable base_url, invalid model_args syntax, unknown task name, missing HF_ALLOW_CODE_EVAL for code tasks, OOM, or missing lm_eval installation ('command not found' in stderr).

Common situations: Benchmarks pointed at a headroom proxy port that isn't up (connection refused in stderr); typo'd task IDs; GPU/CUDA mismatches; lm-eval version changes altering CLI flags; API-key auth failures against the local-chat-completions endpoint.

Related errors

• OPENAI_API_KEY environment variable required
• Error: Memory eval dependencies not installed.
• Error: litellm required for --answer-model. Run: pip install
• Evaluation interrupted.
• Error: Memory eval V2 dependencies not installed.

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/2490b4e846c7fc38. Report an issue: GitHub.