MemPalace/mempalace · error · DaemonError

daemon did not become ready: {last_error}

Error message

daemon did not become ready: {last_error}

What it means

DaemonError raised by start_daemon (mempalace/daemon.py:1353) when the readiness deadline elapsed while the child process is still alive but never answered health checks. The message embeds the last DaemonError from the polling loop (e.g. endpoint missing, token not found, connection refused), which names the actual blocker. The child is killed and reaped before raising.

Source

Thrown at mempalace/daemon.py:1353

        proc = subprocess.Popen(cmd, env=env, **kwargs)
    finally:
        log_fh = kwargs.get("stdout")
        if hasattr(log_fh, "close"):
            log_fh.close()
    try:
        deadline = time.monotonic() + timeout
        last_error = None
        while time.monotonic() < deadline:
            if proc.poll() is not None:
                raise DaemonError(f"daemon exited during startup with code {proc.returncode}")
            try:
                client = DaemonClient(palace_path)
                client.health()
                return client
            except DaemonError as exc:
                last_error = exc
                time.sleep(0.1)
        raise DaemonError(f"daemon did not become ready: {last_error}")
    except BaseException:
        # Readiness failed — don't leak an orphaned detached child holding the
        # port, token, queue, and log handle. Kill and reap it before raising.
        if proc is not None and proc.poll() is None:
            try:
                proc.kill()
                proc.wait()
            except Exception:  # noqa: BLE001 - cleanup best-effort
                pass
        raise
    finally:
        if lock_fh is not None:
            try:
                lock_fh.close()
            except Exception:  # noqa: BLE001 - cleanup best-effort
                pass

View on GitHub (pinned to 06cb6987f0)

Solutions

1. Read the trailing last_error inside the message — it tells you whether startup was stuck on endpoint discovery, auth, or connection.
2. Increase the start_daemon timeout for slow environments (CI, cold caches, heavy backends).
3. Check the daemon log for progress/errors and confirm the palace lock is not held by another process.
4. Retry once after the environment warms up; ensure cleanup already killed the stuck child.

Example fix

# before
client = start_daemon(palace_path, timeout=10)

# after
client = start_daemon(palace_path, timeout=60)  # slow CI / heavy backend import

from mempalace.daemon import DaemonError

try:
    client = start_daemon(palace_path, timeout=10)
except DaemonError as exc:
    if "did not become ready" in str(exc):
        client = start_daemon(palace_path, timeout=60)  # slow start; cleanup killed the stuck child
    else:
        raise

When it happens

Trigger: Slow machine/cold start exceeding the timeout; the daemon bound a different port than endpoint.json advertises; health endpoint blocked; startup deadlock (e.g. waiting on a lock) that keeps the process alive but not serving; extremely short timeouts in tests.

Common situations: CI runners with slow disk where imports of heavy backends (ChromaDB) take seconds; timeout budget too small relative to model/backend initialization; a lock held by a live process making the child wait; first-run setup work inside startup.

Related errors

• daemon token not found for {palace_path}
• writable daemon startup refused: another writer owns local b
• daemon endpoint missing port
• daemon endpoint pid is not alive
• timed out waiting for job {job_id}

AI-assisted analysis of MemPalace/mempalace@06cb6987f0 (2026-08-15). Data as JSON: /api/errors/05c06284d232cf56. Report an issue: GitHub.