headroomlabs-ai/headroom · error · TimeoutError

Timed out waiting for DirectMem0 background writes: {pending

Error message

Timed out waiting for DirectMem0 background writes: {pending_ids}

What it means

TimeoutError raised in DirectMem0Adapter's close/shutdown path while holding _close_lock: after asyncio.wait(tasks, timeout=timeout), background executor-backed writes are still pending. The message lists the offending task ids so you can correlate them with the queued writes that never quiesced.

Source

Thrown at headroom/memory/backends/direct_mem0.py:980

            timeout: Maximum seconds to wait for background writes to finish.

        Raises:
            TimeoutError: If background writes have not quiesced within
                ``timeout``.  Tasks remain tracked and resources remain open so
                callers can retry after the writes finish.
        """
        # Concurrent shutdown callers must observe one lifecycle transition.
        # In particular, a second caller must not detach resources while the
        # first is still waiting for executor-backed writes to quiesce.
        async with self._close_lock:
            if self._background_tasks:
                task_items = list(self._background_tasks.items())
                tasks = [task for _, task in task_items]
                _, pending = await asyncio.wait(tasks, timeout=timeout)

                if pending:
                    pending_ids = [task_id for task_id, task in task_items if task in pending]
                    raise TimeoutError(
                        "Timed out waiting for DirectMem0 background writes: "
                        + ", ".join(pending_ids)
                    )

                for task_id, task in task_items:
                    try:
                        self._task_results[task_id] = {
                            "status": "completed",
                            "result": task.result(),
                        }
                    except Exception as e:
                        self._task_results[task_id] = {
                            "status": "failed",
                            "error": str(e),
                        }
                self._background_tasks.clear()

            resources = [

View on GitHub (pinned to 322425c43b)

Solutions

Re-check service health (Qdrant/Neo4j up? docker compose ps) — the usual cause is writes blocked on a dead connection
Call close() again with a larger timeout: await backend.close(timeout=60)
Drain writes before shutdown by awaiting pending background tasks or checking backend task status first
Inspect the task ids in the message to identify which writes did not finish and whether the data loss matters

Example fix

# before
await backend.close()  # TimeoutError: Timed out waiting for DirectMem0 background writes: task-3, task-7

# after
# give executor-backed writes time to drain
try:
    await backend.close(timeout=60)
except TimeoutError:
    logger.warning('writes still pending; forcing shutdown')
    raise

Defensive patterns

Strategy: retry

Validate before calling

# drain background writes before closing
for task_id, task in list(backend._background_tasks.items()):
    if not task.done():
        logger.info('waiting for background write %s', task_id)
# then close with a generous timeout

Try / catch

try:
    await backend.close(timeout=30)
except TimeoutError as e:
    logger.warning('pending writes on close: %s', e)
    # decide: retry once with a longer window, or force shutdown and report data loss
    await backend.close(timeout=120)

Prevention

Close with an explicit, generous timeout proportional to write backlog
Health-check Qdrant/Neo4j before shutdown so writes do not hang on dead connections
Treat the task ids in the message as the audit list of possibly-lost writes

When it happens

Trigger: Calling close()/shutdown with a short timeout while background memory writes are still executing; Qdrant/Neo4j slow or unreachable so executor writes hang; very large write backlogs exceeding the default wait.

Common situations: Service shutdown under load; a wedged Qdrant connection with no connect-timeout; calling close twice concurrently (handled by the lock) or closing immediately after a burst of saves.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/21b8f1bb495565ab. Report an issue: GitHub.