666ghj/MiroFish · error · TimeoutError

Zep updater drain deadline elapsed before flushing all activ

Error message

Zep updater drain deadline elapsed before flushing all activities

What it means

Raised by the updater's drain routine when the caller-supplied deadline (time.time() >= deadline) is reached before every per-platform activity buffer has been flushed to Zep. It is a cooperative cancellation signal: remaining buffered activities stay in _platform_buffers so the failure is surfaced to SimulationRunner rather than silently dropped.

Source

Thrown at backend/app/services/zep_graph_memory_updater.py:582

            try:
                activity = self._activity_queue.get_nowait()
                platform = activity.platform.lower()
                with self._buffer_lock:
                    if platform not in self._platform_buffers:
                        self._platform_buffers[platform] = []
                    self._platform_buffers[platform].append(activity)
            except Empty:
                break
        
        for platform in list(self._platform_buffers):
            with self._buffer_lock:
                buffer = list(self._platform_buffers.get(platform, []))
            if not buffer:
                continue
            display_name = self._get_platform_display_name(platform)
            logger.info(f"发送{display_name}平台剩余的 {len(buffer)} 条活动")
            if deadline is not None and time.time() >= deadline:
                raise TimeoutError(
                    "Zep updater drain deadline elapsed before flushing all activities"
                )
            try:
                processed_count = self._send_batch_activities(
                    buffer,
                    platform,
                    deadline=deadline,
                )
            except _DrainDeadlineExceeded as error:
                with self._buffer_lock:
                    del self._platform_buffers[platform][:error.processed_count]
                raise TimeoutError(str(error)) from error
            else:
                with self._buffer_lock:
                    del self._platform_buffers[platform][:processed_count]

    def _wait_for_pending_episodes(self, *, deadline: float | None = None) -> None:
        pending = set(self._pending_episode_uuids)

View on GitHub (pinned to b5b53acc57)

Solutions

1. Increase or remove the drain deadline if the data must be flushed (accept longer shutdown)
2. Check the updater stats (total_sent vs buffered counts) to size batches/deadline realistically
3. Address the root cause of slowness: Zep rate limits, network latency, oversized batches
4. Treat the TimeoutError as 'batch incomplete': buffers are preserved, so re-run drain with a fresh deadline instead of resending manually

Example fix

# before
drain(deadline=time.time() + 5)  # too tight for a large backlog

# after
budget = max(30, estimated_batches * avg_batch_seconds * 2)
drain(deadline=time.time() + budget)

remaining = sum(len(b) for b in updater._platform_buffers.values())
if remaining and deadline_soon:
    # extend or re-plan instead of letting drain raise
    deadline = time.time() + new_budget

try:
    updater.drain(deadline=deadline)
except TimeoutError:
    # buffers are preserved; retry drain with a fresh deadline
    updater.drain(deadline=time.time() + EXTENDED_BUDGET)

When it happens

Trigger: Calling the drain/flush path with a deadline after slow Zep writes have consumed most of the budget: by the time the loop reaches a platform whose buffer is non-empty, time.time() >= deadline. Large backlogs, slow network, or an unrealistically tight deadline trigger it before _send_batch_activities is even attempted.

Common situations: Simulation shutdown with a fixed timeout, Zep Cloud latency spikes or rate limiting (each retry burns deadline budget), or a burst of activity that filled buffers faster than batches could be sent.

Related errors

• Zep batch {submission.batch_id} did not finish within {timeo
• Zep episode processing timed out with {len(pending_episodes)
• 模拟仍在停止中，图谱写入未在 {wait_timeout:.0f}s 内完成
• Zep updater worker did not stop within {join_timeout:.0f}s
• Zep updater drain deadline elapsed

AI-assisted analysis of 666ghj/MiroFish@b5b53acc57 (2026-08-14). Data as JSON: /api/errors/32e4d11399a317a9. Report an issue: GitHub.