MemPalace/mempalace · warning · DaemonError

timed out waiting for job {job_id}

Error message

timed out waiting for job {job_id}

What it means

DaemonError raised by the client's job-wait loop (mempalace/daemon.py:1217): the job had not reached a terminal state before the caller's timeout deadline (checked with time.monotonic after each 0.2s poll). The job itself is not cancelled — it continues in the daemon queue; only the local wait aborts.

Source

Thrown at mempalace/daemon.py:1217

        self,
        job_id: str,
        *,
        timeout: float = DEFAULT_WAIT_TIMEOUT,
        stop_on_lock_deferral: bool = False,
    ) -> dict[str, Any]:
        deadline = time.monotonic() + timeout
        while True:
            job = self.get_job(job_id)
            if job["state"] in TERMINAL_STATES:
                return job
            # A job parked behind the palace lock never becomes terminal on its
            # own, so an interactive caller must be able to stop here instead of
            # waiting out the holder (#2014). Background callers keep the old
            # behaviour and simply wait.
            if stop_on_lock_deferral and job_deferred_by_lock(job):
                return job
            if time.monotonic() >= deadline:
                raise DaemonError(f"timed out waiting for job {job_id}")
            time.sleep(0.2)

    def shutdown(self) -> dict[str, Any]:
        return self.request("POST", "/shutdown", {})


def get_client_if_running(palace_path: str, *, health_timeout: float = 5.0) -> DaemonClient | None:
    # health_timeout bounds the liveness probe. Hook callers (subject to the
    # ~500ms hook budget) pass a short value via HOOK_PROBE_TIMEOUT so a wedged
    # daemon — endpoint present, HTTP server not answering — can't stall the
    # hook for the default 5s before it falls back to the direct path.
    try:
        client = DaemonClient(palace_path)
        client.health(timeout=health_timeout)
        return client
    except DaemonError:
        return None

View on GitHub (pinned to 06cb6987f0)

Solutions

  1. Increase the timeout for genuinely long jobs (indexing, embedding over large chunks).
  2. If the job may be parked behind the palace lock, use the interactive path with stop_on_lock_deferral=True to return the deferred job instead of timing out.
  3. Check client.get_job(job_id) / counts() to see queue depth and whether the job is running, queued, or deferred; free the lock holder if deferred.
  4. Treat the timeout as 'not done yet', not failure: poll again later or let the daemon finish in the background.

Example fix

# before
job = client.wait_for_job(job_id, timeout=5)

# after
job = client.wait_for_job(job_id, timeout=120, stop_on_lock_deferral=True)
if not is_terminal(job):
    # resume later; job still queued in daemon
Defensive patterns

Strategy: retry

Try / catch

from mempalace.daemon import DaemonError

try:
    job = client.wait_for_job(job_id, timeout=120, stop_on_lock_deferral=True)
except DaemonError as exc:
    if "timed out" in str(exc):
        job = client.get_job(job_id)  # still queued/running; check later, don't fail
    else:
        raise

Prevention

When it happens

Trigger: Waiting on a long-running indexing/embedding job with a short timeout; the queue is backed up behind higher-priority jobs; the job is parked waiting on the palace lock held by another writer (interactive callers can pass stop_on_lock_deferral to return early instead); a wedged worker.

Common situations: Hook budgets (~500ms) forcing tiny timeouts on genuinely slow jobs; bulk ingest saturating the single worker; another process holding the palace lock so the job defers indefinitely (see #2014 referenced in the source); default timeouts too small for cold-start model loads.

Understand the failure class

Related errors

AI-assisted analysis of MemPalace/mempalace@06cb6987f0 (2026-08-15). Data as JSON: /api/errors/5e69b0b9f4dbc301. Report an issue: GitHub.