{"record":{"id":"2490b4e846c7fc38","repo":"headroomlabs-ai/headroom","slug":"lm-eval-failed-result-stderr","errorCode":null,"errorMessage":"lm-eval failed: {result.stderr}","messagePattern":"lm-eval failed: (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"headroom/evals/comprehensive_benchmark.py","lineNumber":286,"sourceCode":"\n    # Run lm-eval\n    start_time = time.time()\n    env = {\n        **os.environ,\n        \"TOKENIZERS_PARALLELISM\": \"false\",\n        \"HF_ALLOW_CODE_EVAL\": \"1\",  # Required for humaneval/mbpp tasks\n    }\n    result = run(\n        cmd,\n        capture_output=True,\n        text=True,\n        env=env,\n    )\n    duration = time.time() - start_time\n\n    if result.returncode != 0:\n        logger.error(f\"lm-eval failed: {result.stderr}\")\n        raise RuntimeError(f\"lm-eval failed: {result.stderr}\")\n\n    # Load results - lm-eval creates a directory structure with timestamped files\n    results_dir: Path = Path(output_path) if output_path else Path(\".\")\n\n    # Find results_*.json in the output directory (lm-eval uses timestamped filenames)\n    results_file: Path | None = None\n    if results_dir.is_dir():\n        # Look for results_*.json files\n        for f in sorted(results_dir.glob(\"**/results_*.json\"), reverse=True):\n            results_file = f\n            break\n\n    if results_file is None or not results_file.exists():\n        # Parse results from stdout as fallback\n        logger.warning(\"No results file found, parsing from stdout\")\n        return {\"results\": {}, \"_duration_seconds\": duration, \"_stdout\": result.stdout}\n\n    with open(results_file) as fp:","sourceCodeStart":268,"sourceCodeEnd":304,"githubUrl":"https://github.com/headroomlabs-ai/headroom/blob/322425c43bffde1ed0b64fecf3cf5951565dd82b/headroom/evals/comprehensive_benchmark.py#L268-L304","documentation":"Raised after run_lm_eval() shells out to the lm-eval CLI (subprocess.run with HF_ALLOW_CODEEval=1 for humaneval/mbpp): a non-zero return code triggers a logged error plus RuntimeError carrying the child process's stderr. The message 'lm-eval failed' therefore wraps arbitrary downstream failures — model connection errors, missing tasks, CUDA issues, bad model_args — and stderr is the real diagnostic.","triggerScenarios":"run_lm_eval(model=..., model_args=..., tasks=..., base_url=...) where the lm-eval CLI exits non-zero: unreachable base_url, invalid model_args syntax, unknown task name, missing HF_ALLOW_CODE_EVAL for code tasks, OOM, or missing lm_eval installation ('command not found' in stderr).","commonSituations":"Benchmarks pointed at a headroom proxy port that isn't up (connection refused in stderr); typo'd task IDs; GPU/CUDA mismatches; lm-eval version changes altering CLI flags; API-key auth failures against the local-chat-completions endpoint.","solutions":["Read the embedded stderr first — it names the actual failing component; the headroom-level message adds no extra information","Verify the prerequisites the subprocess assumes: lm-eval installed (`lm_eval --version`), model endpoint reachable at base_url, tasks spelled per the lm-eval task registry","Reproduce manually with the same command and env (HF_ALLOW_CODE_EVAL=1) to iterate faster than through headroom","For code tasks, confirm humaneval/mbpp execution flags are present since this runner sets them specifically for that purpose"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"import shutil\n\nif shutil.which(\"lm_eval\") is None:\n    raise SystemExit(\"lm-eval CLI not installed — pip install lm-eval before benchmarking\")","typeGuard":null,"tryCatchPattern":"try:\n    raw = run_lm_eval(model=..., tasks=tasks, base_url=base_url, ...)\nexcept RuntimeError as e:\n    raise SystemExit(f\"lm-eval subprocess failed — see stderr: {e}\") from e","preventionTips":["Always surface the embedded stderr — the headroom message alone has no root cause","Pre-flight the model endpoint (curl base_url) and task names before the expensive run","Reproduce the lm-eval command manually with HF_ALLOW_CODE_EVAL=1 when iterating on failures"],"tags":["subprocess","benchmark","lm-eval","evals","harness"],"backgroundTag":null,"analyzedSha":"322425c43bffde1ed0b64fecf3cf5951565dd82b","analyzedAt":"2026-08-15T01:03:05.481Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}