{"record":{"id":"21b8f1bb495565ab","repo":"headroomlabs-ai/headroom","slug":"timed-out-waiting-for-directmem0-background-writes","errorCode":null,"errorMessage":"Timed out waiting for DirectMem0 background writes: {pending_ids}","messagePattern":"Timed out waiting for DirectMem0 background writes: (.+?)","errorType":"exception","errorClass":"TimeoutError","httpStatus":null,"severity":"error","filePath":"headroom/memory/backends/direct_mem0.py","lineNumber":980,"sourceCode":"            timeout: Maximum seconds to wait for background writes to finish.\n\n        Raises:\n            TimeoutError: If background writes have not quiesced within\n                ``timeout``.  Tasks remain tracked and resources remain open so\n                callers can retry after the writes finish.\n        \"\"\"\n        # Concurrent shutdown callers must observe one lifecycle transition.\n        # In particular, a second caller must not detach resources while the\n        # first is still waiting for executor-backed writes to quiesce.\n        async with self._close_lock:\n            if self._background_tasks:\n                task_items = list(self._background_tasks.items())\n                tasks = [task for _, task in task_items]\n                _, pending = await asyncio.wait(tasks, timeout=timeout)\n\n                if pending:\n                    pending_ids = [task_id for task_id, task in task_items if task in pending]\n                    raise TimeoutError(\n                        \"Timed out waiting for DirectMem0 background writes: \"\n                        + \", \".join(pending_ids)\n                    )\n\n                for task_id, task in task_items:\n                    try:\n                        self._task_results[task_id] = {\n                            \"status\": \"completed\",\n                            \"result\": task.result(),\n                        }\n                    except Exception as e:\n                        self._task_results[task_id] = {\n                            \"status\": \"failed\",\n                            \"error\": str(e),\n                        }\n                self._background_tasks.clear()\n\n            resources = [","sourceCodeStart":962,"sourceCodeEnd":998,"githubUrl":"https://github.com/headroomlabs-ai/headroom/blob/322425c43bffde1ed0b64fecf3cf5951565dd82b/headroom/memory/backends/direct_mem0.py#L962-L998","documentation":"TimeoutError raised in DirectMem0Adapter's close/shutdown path while holding _close_lock: after asyncio.wait(tasks, timeout=timeout), background executor-backed writes are still pending. The message lists the offending task ids so you can correlate them with the queued writes that never quiesced.","triggerScenarios":"Calling close()/shutdown with a short timeout while background memory writes are still executing; Qdrant/Neo4j slow or unreachable so executor writes hang; very large write backlogs exceeding the default wait.","commonSituations":"Service shutdown under load; a wedged Qdrant connection with no connect-timeout; calling close twice concurrently (handled by the lock) or closing immediately after a burst of saves.","solutions":["Re-check service health (Qdrant/Neo4j up? docker compose ps) — the usual cause is writes blocked on a dead connection","Call close() again with a larger timeout: await backend.close(timeout=60)","Drain writes before shutdown by awaiting pending background tasks or checking backend task status first","Inspect the task ids in the message to identify which writes did not finish and whether the data loss matters"],"exampleFix":"# before\nawait backend.close()  # TimeoutError: Timed out waiting for DirectMem0 background writes: task-3, task-7\n\n# after\n# give executor-backed writes time to drain\ntry:\n    await backend.close(timeout=60)\nexcept TimeoutError:\n    logger.warning('writes still pending; forcing shutdown')\n    raise","handlingStrategy":"retry","validationCode":"# drain background writes before closing\nfor task_id, task in list(backend._background_tasks.items()):\n    if not task.done():\n        logger.info('waiting for background write %s', task_id)\n# then close with a generous timeout","typeGuard":null,"tryCatchPattern":"try:\n    await backend.close(timeout=30)\nexcept TimeoutError as e:\n    logger.warning('pending writes on close: %s', e)\n    # decide: retry once with a longer window, or force shutdown and report data loss\n    await backend.close(timeout=120)","preventionTips":["Close with an explicit, generous timeout proportional to write backlog","Health-check Qdrant/Neo4j before shutdown so writes do not hang on dead connections","Treat the task ids in the message as the audit list of possibly-lost writes"],"tags":["python","asyncio","timeout","shutdown","background-tasks","mem0"],"backgroundTag":null,"analyzedSha":"322425c43bffde1ed0b64fecf3cf5951565dd82b","analyzedAt":"2026-08-15T01:03:05.481Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}