{"record":{"id":"521c6cceebdc5569","repo":"calesthio/OpenMontage","slug":"backlot-server-did-not-become-healthy","errorCode":null,"errorMessage":"Backlot server did not become healthy","messagePattern":"Backlot server did not become healthy","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"scripts/backlot_visual_eval.py","lineNumber":116,"sourceCode":"def start_server() -> subprocess.Popen:\n    env = dict(os.environ)\n    env[\"OPENMONTAGE_PROJECTS_DIR\"] = str(STAGE_DIR)\n    server = subprocess.Popen(\n        [sys.executable, \"-m\", \"backlot\", \"serve\", \"--port\", str(PORT)],\n        cwd=REPO_ROOT,\n        env=env,\n        stdout=subprocess.DEVNULL,\n        stderr=subprocess.DEVNULL,\n    )\n    deadline = time.time() + 20\n    while time.time() < deadline:\n        try:\n            with urllib.request.urlopen(f\"http://127.0.0.1:{PORT}/api/health\", timeout=1):\n                return server\n        except Exception:\n            time.sleep(0.3)\n    server.terminate()\n    raise RuntimeError(\"Backlot server did not become healthy\")\n\n\ndef capture_screenshot(url: str, output: Path, width: int, height: int, wait_ms: int) -> None:\n    output.parent.mkdir(parents=True, exist_ok=True)\n    subprocess.run(\n        [\n            \"npx\",\n            \"playwright\",\n            \"screenshot\",\n            \"--viewport-size\",\n            f\"{width},{height}\",\n            \"--wait-for-timeout\",\n            str(wait_ms),\n            url,\n            str(output),\n        ],\n        cwd=REPO_ROOT,\n        check=True,","sourceCodeStart":98,"sourceCodeEnd":134,"githubUrl":"https://github.com/calesthio/OpenMontage/blob/95e1c3d0ab93482159818560f6a8c8e866b9139f/scripts/backlot_visual_eval.py#L98-L134","documentation":"Raised by scripts/backlot_visual_eval.py when the locally spawned Backlot server process fails to answer GET /api/health on 127.0.0.1:{PORT} within a 20-second deadline (polled every 0.3s). The server subprocess is started with stdout/stderr discarded to DEVNULL, so the health timeout is often the only visible symptom — the real cause (crash on startup, port already in use, missing dependency) is hidden in the discarded output. The server is terminated before the error raises.","triggerScenarios":"Running the visual eval when the Backlot server crashes at boot (missing dependency, bad config), when PORT is already occupied by another process so the new server fails to bind, or when the machine is so loaded that startup exceeds 20s.","commonSituations":"Port collision with a dev server left running; first run before server dependencies are installed; a refactor that broke server startup; slow CI runners hitting the fixed 20s deadline.","solutions":["Free the port: stop any process already listening on PORT (the eval spawns its own server and expects it exclusive).","Reproduce the startup failure manually by launching the server with visible output to see the actual error.","If startup is just slow, raise the 20s deadline in scripts/backlot_visual_eval.py — and for debugging, temporarily stop discarding stderr."],"exampleFix":"# debugging — before\nstdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL\n\n# debugging — after (see the real boot error)\nstdout=None, stderr=None  # or log to a file","handlingStrategy":"retry","validationCode":"import socket\n\ndef port_is_free(port: int) -> bool:\n    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:\n        return s.connect_ex((\"127.0.0.1\", port)) != 0 or False\n# simpler: check nothing is listening\ndef nothing_listening(port: int) -> bool:\n    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:\n        s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n        try:\n            s.bind((\"127.0.0.1\", port))\n            return True\n        except OSError:\n            return False","typeGuard":null,"tryCatchPattern":"try:\n    run_interaction_smoke()\nexcept RuntimeError as e:\n    if \"did not become healthy\" in str(e):\n        # port conflict or boot crash — free the port / boot the server manually to see logs, then retry once\n        log.error(\"Backlot server failed health check; check port %d and server startup logs\", PORT)\n    raise","preventionTips":["Kill processes occupying the eval port before running the script.","Boot the server once manually with visible logs after any server dependency change.","When debugging, redirect the subprocess stderr to a file instead of DEVNULL."],"tags":["server","health-check","timeout","port","eval"],"backgroundTag":null,"analyzedSha":"95e1c3d0ab93482159818560f6a8c8e866b9139f","analyzedAt":"2026-08-15T06:31:20.014Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}