666ghj/MiroFish · error · TimeoutError

Zep simulation ingestion timed out with {len(pending)} episo

Error message

Zep simulation ingestion timed out with {len(pending)} episode(s) pending

What it means

Raised by _wait_for_pending_episodes when polling Zep's asynchronous ingestion: episodes accepted by graph.add are polled via client.graph.episode.get(uuid_=...) until episode.processed is True, and if the deadline (default ZEP_INGESTION_WAIT_TIMEOUT_SECONDS = 600s, polled every 3s) passes with episodes still unprocessed, this TimeoutError fires with the pending count.

Source

Thrown at backend/app/services/zep_graph_memory_updater.py:608

                )
            except _DrainDeadlineExceeded as error:
                with self._buffer_lock:
                    del self._platform_buffers[platform][:error.processed_count]
                raise TimeoutError(str(error)) from error
            else:
                with self._buffer_lock:
                    del self._platform_buffers[platform][:processed_count]

    def _wait_for_pending_episodes(self, *, deadline: float | None = None) -> None:
        pending = set(self._pending_episode_uuids)
        if not pending:
            return

        if deadline is None:
            deadline = time.time() + ZEP_INGESTION_WAIT_TIMEOUT_SECONDS
        while pending:
            if time.time() >= deadline:
                raise TimeoutError(
                    f"Zep simulation ingestion timed out with {len(pending)} "
                    "episode(s) pending"
                )
            for episode_uuid in list(pending):
                episode = call_zep_read_with_retry(
                    lambda: self.client.graph.episode.get(uuid_=episode_uuid),
                    operation_name=f"poll simulation episode {episode_uuid}",
                )
                if getattr(episode, "processed", False):
                    pending.remove(episode_uuid)
            if pending:
                time.sleep(3)
        self._pending_episode_uuids = []
    
    def get_stats(self) -> Dict[str, Any]:
        """获取统计信息"""
        with self._buffer_lock:
            buffer_sizes = {p: len(b) for p, b in self._platform_buffers.items()}

View on GitHub (pinned to b5b53acc57)

Solutions

Retry the wait with a fresh deadline — episodes may still complete; re-poll the same UUIDs before assuming failure
Increase ZEP_INGESTION_WAIT_TIMEOUT_SECONDS for large simulations
Check Zep Cloud status/queue latency; reduce batch sizes so extraction finishes faster
If an episode never becomes processed, fetch it via graph.episode.get and inspect its state for server-side errors
Send activities incrementally during the simulation instead of a large end-of-run flush

Example fix

# before
ZEP_INGESTION_WAIT_TIMEOUT_SECONDS = 600

# after (sized to worst observed ingestion latency)
ZEP_INGESTION_WAIT_TIMEOUT_SECONDS = 900

Defensive patterns

Strategy: retry

Try / catch

try:
    updater._wait_for_pending_episodes(deadline=d)
except TimeoutError:
    # episodes may still complete later; re-poll before declaring failure
    updater._wait_for_pending_episodes(deadline=time.time() + EXTRA_WAIT)

Prevention

Increase ZEP_INGESTION_WAIT_TIMEOUT_SECONDS for large simulations
Flush activities throughout the run so ingestion overlaps with simulation time
Track per-episode UUIDs so a retry polls only genuinely pending episodes

When it happens

Trigger: After sending episodes, polling loop hits time.time() >= deadline while at least one episode still has processed=False. Triggered by heavy Zep Cloud ingestion queues, large batches of extracted facts, or Zep-side outages that leave episodes perpetually unprocessed.

Common situations: Simulations that push hundreds of activities near the end of the ingestion window; Zep Cloud load or maintenance slowing extraction; episodes rejected server-side without being marked processed. The 600s constant may be too short for very large knowledge-graph updates.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of 666ghj/MiroFish@b5b53acc57 (2026-08-14). Data as JSON: /api/errors/277a6fcd4e302b3c. Report an issue: GitHub.