calesthio/OpenMontage · error · RuntimeError

Backlot server did not become healthy

Error message

Backlot server did not become healthy

What it means

Raised by scripts/backlot_visual_eval.py when the locally spawned Backlot server process fails to answer GET /api/health on 127.0.0.1:{PORT} within a 20-second deadline (polled every 0.3s). The server subprocess is started with stdout/stderr discarded to DEVNULL, so the health timeout is often the only visible symptom — the real cause (crash on startup, port already in use, missing dependency) is hidden in the discarded output. The server is terminated before the error raises.

Source

Thrown at scripts/backlot_visual_eval.py:116

def start_server() -> subprocess.Popen:
    env = dict(os.environ)
    env["OPENMONTAGE_PROJECTS_DIR"] = str(STAGE_DIR)
    server = subprocess.Popen(
        [sys.executable, "-m", "backlot", "serve", "--port", str(PORT)],
        cwd=REPO_ROOT,
        env=env,
        stdout=subprocess.DEVNULL,
        stderr=subprocess.DEVNULL,
    )
    deadline = time.time() + 20
    while time.time() < deadline:
        try:
            with urllib.request.urlopen(f"http://127.0.0.1:{PORT}/api/health", timeout=1):
                return server
        except Exception:
            time.sleep(0.3)
    server.terminate()
    raise RuntimeError("Backlot server did not become healthy")


def capture_screenshot(url: str, output: Path, width: int, height: int, wait_ms: int) -> None:
    output.parent.mkdir(parents=True, exist_ok=True)
    subprocess.run(
        [
            "npx",
            "playwright",
            "screenshot",
            "--viewport-size",
            f"{width},{height}",
            "--wait-for-timeout",
            str(wait_ms),
            url,
            str(output),
        ],
        cwd=REPO_ROOT,
        check=True,

View on GitHub (pinned to 95e1c3d0ab)

Solutions

  1. Free the port: stop any process already listening on PORT (the eval spawns its own server and expects it exclusive).
  2. Reproduce the startup failure manually by launching the server with visible output to see the actual error.
  3. If startup is just slow, raise the 20s deadline in scripts/backlot_visual_eval.py — and for debugging, temporarily stop discarding stderr.

Example fix

# debugging — before
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL

# debugging — after (see the real boot error)
stdout=None, stderr=None  # or log to a file
Defensive patterns

Strategy: retry

Validate before calling

import socket

def port_is_free(port: int) -> bool:
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        return s.connect_ex(("127.0.0.1", port)) != 0 or False
# simpler: check nothing is listening
def nothing_listening(port: int) -> bool:
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        try:
            s.bind(("127.0.0.1", port))
            return True
        except OSError:
            return False

Try / catch

try:
    run_interaction_smoke()
except RuntimeError as e:
    if "did not become healthy" in str(e):
        # port conflict or boot crash — free the port / boot the server manually to see logs, then retry once
        log.error("Backlot server failed health check; check port %d and server startup logs", PORT)
    raise

Prevention

When it happens

Trigger: Running the visual eval when the Backlot server crashes at boot (missing dependency, bad config), when PORT is already occupied by another process so the new server fails to bind, or when the machine is so loaded that startup exceeds 20s.

Common situations: Port collision with a dev server left running; first run before server dependencies are installed; a refactor that broke server startup; slow CI runners hitting the fixed 20s deadline.

Related errors

AI-assisted analysis of calesthio/OpenMontage@95e1c3d0ab (2026-08-15). Data as JSON: /api/errors/521c6cceebdc5569. Report an issue: GitHub.