666ghj/MiroFish · error · TimeoutError

Zep updater drain deadline elapsed

Error message

Zep updater drain deadline elapsed

What it means

Raised when _send_batch_activities internally aborts with _DrainDeadlineExceeded: the batch send ran out of deadline partway through. The handler first trims the platform buffer by error.processed_count (activities already accepted are removed) and then re-raises TimeoutError, so a resume re-sends only the unsent tail — no duplicates, no data loss.

Source

Thrown at backend/app/services/zep_graph_memory_updater.py:594

                buffer = list(self._platform_buffers.get(platform, []))
            if not buffer:
                continue
            display_name = self._get_platform_display_name(platform)
            logger.info(f"发送{display_name}平台剩余的 {len(buffer)} 条活动")
            if deadline is not None and time.time() >= deadline:
                raise TimeoutError(
                    "Zep updater drain deadline elapsed before flushing all activities"
                )
            try:
                processed_count = self._send_batch_activities(
                    buffer,
                    platform,
                    deadline=deadline,
                )
            except _DrainDeadlineExceeded as error:
                with self._buffer_lock:
                    del self._platform_buffers[platform][:error.processed_count]
                raise TimeoutError(str(error)) from error
            else:
                with self._buffer_lock:
                    del self._platform_buffers[platform][:processed_count]

    def _wait_for_pending_episodes(self, *, deadline: float | None = None) -> None:
        pending = set(self._pending_episode_uuids)
        if not pending:
            return

        if deadline is None:
            deadline = time.time() + ZEP_INGESTION_WAIT_TIMEOUT_SECONDS
        while pending:
            if time.time() >= deadline:
                raise TimeoutError(
                    f"Zep simulation ingestion timed out with {len(pending)} "
                    "episode(s) pending"
                )
            for episode_uuid in list(pending):

View on GitHub (pinned to b5b53acc57)

Solutions

1. Catch TimeoutError at the SimulationRunner level and re-invoke drain with a new deadline — processed items were already trimmed from the buffer
2. Reduce batch size so each _send_batch_activities call fits comfortably inside the remaining budget
3. Scale the deadline to backlog size instead of using a constant
4. Investigate Zep throughput (rate limits, network) if deadlines are regularly exceeded

Example fix

# before
try:
    updater.drain(deadline=deadline)
except TimeoutError:
    abort_simulation()

# after
try:
    updater.drain(deadline=deadline)
except TimeoutError:
    deadline = time.time() + EXTENDED_BUDGET
    updater.drain(deadline=deadline)  # buffer was trimmed; only unsent tail remains

try:
    processed = updater._send_batch_activities(buffer, platform, deadline=deadline)
except _DrainDeadlineExceeded as e:
    # buffer already trimmed to processed_count; only unsent tail remains
    retry_later(remaining=buffer[e.processed_count:])

When it happens

Trigger: A platform's buffered batch is large enough that sending it crosses the deadline mid-batch; _send_batch_activities detects the deadline between chunked writes and raises _DrainDeadlineExceeded carrying the count of already-processed activities.

Common situations: End-of-simulation flush with a deadline sized for smaller batches, slow Zep ingestion throughput, or per-chunk retries (call_zep_read_with_retry style backoff) eating the remaining budget.

Related errors

• Persisted Zep batch does not match the current graph input
• Zep batch {submission.batch_id} did not finish within {timeo
• Zep episode processing timed out with {len(pending_episodes)
• 模拟仍在停止中，图谱写入未在 {wait_timeout:.0f}s 内完成
• Zep图谱写入未完整完成: {error}

AI-assisted analysis of 666ghj/MiroFish@b5b53acc57 (2026-08-14). Data as JSON: /api/errors/3b3daf033ac03498. Report an issue: GitHub.