666ghj/MiroFish · error · TimeoutError

Zep updater worker did not stop within {join_timeout:.0f}s

Error message

Zep updater worker did not stop within {join_timeout:.0f}s

What it means

stop() on the updater drains its worker thread within a deadline derived from ZEP_INGESTION_WAIT_TIMEOUT_SECONDS; if the worker thread is still alive after join_timeout seconds it raises TimeoutError('Zep updater worker did not stop within Ns'). The worker is stuck — typically blocked in a Zep HTTP call or retry backoff — so orderly shutdown cannot be guaranteed.

Source

Thrown at backend/app/services/zep_graph_memory_updater.py:326

            name=f"ZepMemoryUpdater-{self.graph_id[:8]}"
        )
        self._worker_thread.start()
        logger.info(f"ZepGraphMemoryUpdater 已启动: graph_id={self.graph_id}")
    
    def stop(self):
        """Drain the worker, flush tail events, and wait for Cloud ingestion."""
        deadline = time.time() + ZEP_INGESTION_WAIT_TIMEOUT_SECONDS
        # Serialize the accepting->closed transition with add_activity's
        # check+enqueue operation. This closes the small race where a producer
        # could enqueue after both the worker and final flush had exited.
        with self._acceptance_lock:
            self._running = False

        if self._worker_thread and self._worker_thread.is_alive():
            join_timeout = max(0.0, deadline - time.time())
            self._worker_thread.join(timeout=join_timeout)
            if self._worker_thread.is_alive():
                raise TimeoutError(
                    f"Zep updater worker did not stop within {join_timeout:.0f}s"
                )

        # The worker has drained the queue. Only now is it safe to flush
        # buffers; doing this before join loses an item already dequeued by the
        # worker but not yet buffered.
        self._flush_remaining(deadline=deadline)

        if self._failed_batches:
            raise RuntimeError(
                f"{len(self._failed_batches)} Zep activity batch(es) failed; "
                "simulation graph ingestion is incomplete"
            )

        self._wait_for_pending_episodes(deadline=deadline)
        
        logger.info(f"ZepGraphMemoryUpdater 已停止: graph_id={self.graph_id}, "
                   f"total_activities={self._total_activities}, "

View on GitHub (pinned to b5b53acc57)

Solutions

Retry stop() after checking Zep API health — the worker may finish its in-flight request and a second stop can complete.
Increase ZEP_INGESTION_WAIT_TIMEOUT_SECONDS (and ZEP_HTTP_REQUEST_TIMEOUT_SECONDS) to match your batch sizes and network conditions.
If permanently stuck, capture thread dump/logs for the worker, then abandon the thread (daemon) and reconcile the graph later — expect the 'N batch(es) failed' RuntimeError or partial ingestion.

Example fix

# before
updater.stop()  # TimeoutError: worker did not stop within 60s

# after
try:
    updater.stop()
except TimeoutError:
    logger.warning("zep worker stuck; waiting one more cycle")
    time.sleep(30)
    updater.stop()  # second attempt after in-flight HTTP completes

Defensive patterns

Strategy: retry

Try / catch

try:
    updater.stop()
except TimeoutError:
    time.sleep(30)
    updater.stop()  # second attempt; if it still fails, escalate with logs

Prevention

Tune ZEP_INGESTION_WAIT_TIMEOUT_SECONDS to worst-case batch drain time.
Monitor Zep API latency during runs; slow API = slow stop.
Keep per-batch durations logged so the deadline can be set from data.

When it happens

Trigger: Calling stop() while the worker is mid-batch against a slow/hung Zep API; retry backoff (initial_delay 2.0s, 3 retries per call in the reader/updater pattern) stacking beyond the deadline; very large queued backlog exceeding the ingestion wait budget.

Common situations: Zep Cloud latency spikes or outages at simulation stop; on-prem proxy stalling connections; simulations with heavy activity volume so drain time legitimately exceeds the default budget.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of 666ghj/MiroFish@b5b53acc57 (2026-08-14). Data as JSON: /api/errors/4b6098ca4bb75216. Report an issue: GitHub.