{"record":{"id":"5e69b0b9f4dbc301","repo":"MemPalace/mempalace","slug":"timed-out-waiting-for-job-job-id","errorCode":null,"errorMessage":"timed out waiting for job {job_id}","messagePattern":"timed out waiting for job (.+?)","errorType":"exception","errorClass":"DaemonError","httpStatus":null,"severity":"warning","filePath":"mempalace/daemon.py","lineNumber":1217,"sourceCode":"        self,\n        job_id: str,\n        *,\n        timeout: float = DEFAULT_WAIT_TIMEOUT,\n        stop_on_lock_deferral: bool = False,\n    ) -> dict[str, Any]:\n        deadline = time.monotonic() + timeout\n        while True:\n            job = self.get_job(job_id)\n            if job[\"state\"] in TERMINAL_STATES:\n                return job\n            # A job parked behind the palace lock never becomes terminal on its\n            # own, so an interactive caller must be able to stop here instead of\n            # waiting out the holder (#2014). Background callers keep the old\n            # behaviour and simply wait.\n            if stop_on_lock_deferral and job_deferred_by_lock(job):\n                return job\n            if time.monotonic() >= deadline:\n                raise DaemonError(f\"timed out waiting for job {job_id}\")\n            time.sleep(0.2)\n\n    def shutdown(self) -> dict[str, Any]:\n        return self.request(\"POST\", \"/shutdown\", {})\n\n\ndef get_client_if_running(palace_path: str, *, health_timeout: float = 5.0) -> DaemonClient | None:\n    # health_timeout bounds the liveness probe. Hook callers (subject to the\n    # ~500ms hook budget) pass a short value via HOOK_PROBE_TIMEOUT so a wedged\n    # daemon — endpoint present, HTTP server not answering — can't stall the\n    # hook for the default 5s before it falls back to the direct path.\n    try:\n        client = DaemonClient(palace_path)\n        client.health(timeout=health_timeout)\n        return client\n    except DaemonError:\n        return None\n","sourceCodeStart":1199,"sourceCodeEnd":1235,"githubUrl":"https://github.com/MemPalace/mempalace/blob/06cb6987f02610784fefbad4b2bd5d026d164ba6/mempalace/daemon.py#L1199-L1235","documentation":"DaemonError raised by the client's job-wait loop (mempalace/daemon.py:1217): the job had not reached a terminal state before the caller's timeout deadline (checked with time.monotonic after each 0.2s poll). The job itself is not cancelled — it continues in the daemon queue; only the local wait aborts.","triggerScenarios":"Waiting on a long-running indexing/embedding job with a short timeout; the queue is backed up behind higher-priority jobs; the job is parked waiting on the palace lock held by another writer (interactive callers can pass stop_on_lock_deferral to return early instead); a wedged worker.","commonSituations":"Hook budgets (~500ms) forcing tiny timeouts on genuinely slow jobs; bulk ingest saturating the single worker; another process holding the palace lock so the job defers indefinitely (see #2014 referenced in the source); default timeouts too small for cold-start model loads.","solutions":["Increase the timeout for genuinely long jobs (indexing, embedding over large chunks).","If the job may be parked behind the palace lock, use the interactive path with stop_on_lock_deferral=True to return the deferred job instead of timing out.","Check client.get_job(job_id) / counts() to see queue depth and whether the job is running, queued, or deferred; free the lock holder if deferred.","Treat the timeout as 'not done yet', not failure: poll again later or let the daemon finish in the background."],"exampleFix":"# before\njob = client.wait_for_job(job_id, timeout=5)\n\n# after\njob = client.wait_for_job(job_id, timeout=120, stop_on_lock_deferral=True)\nif not is_terminal(job):\n    # resume later; job still queued in daemon","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"from mempalace.daemon import DaemonError\n\ntry:\n    job = client.wait_for_job(job_id, timeout=120, stop_on_lock_deferral=True)\nexcept DaemonError as exc:\n    if \"timed out\" in str(exc):\n        job = client.get_job(job_id)  # still queued/running; check later, don't fail\n    else:\n        raise","preventionTips":["Size timeouts to the job kind (indexing/embedding >> save).","Use stop_on_lock_deferral for interactive callers so lock parking returns early.","Treat timeout as 'pending', poll again or inspect queue counts before resubmitting."],"tags":["daemon","jobs","timeout","polling","concurrency"],"backgroundTag":null,"analyzedSha":"06cb6987f02610784fefbad4b2bd5d026d164ba6","analyzedAt":"2026-08-15T03:03:36.213Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}