666ghj/MiroFish · warning · SimulationStopPending

模拟仍在停止中，图谱写入未在 {wait_timeout:.0f}s 内完成

Error message

模拟仍在停止中，图谱写入未在 {wait_timeout:.0f}s 内完成

What it means

SimulationStopPending (a TimeoutError subclass) is raised when stop_simulation has signaled termination but the background monitor thread still owns finalization after waiting wait_timeout = max(30, ZEP_INGESTION_WAIT_TIMEOUT + ZEP_HTTP_REQUEST_TIMEOUT + 5) seconds — typically because Zep graph-memory HTTP ingestion is slow. The runner deliberately does NOT contend for the finalization lock; observable state stays STOPPING and polling will eventually show STOPPED or FAILED.

Source

Thrown at backend/app/services/simulation_runner.py:1038

            not retrying_finalization
            and
            monitor is not None
            and monitor is not threading.current_thread()
            and monitor.is_alive()
        ):
            wait_timeout = max(
                30.0,
                ZEP_INGESTION_WAIT_TIMEOUT_SECONDS
                + ZEP_HTTP_REQUEST_TIMEOUT_SECONDS
                + 5,
            )
            monitor.join(timeout=wait_timeout)
            if monitor.is_alive():
                # The monitor still owns finalization and may be inside one
                # bounded HTTP request. Do not block on or overwrite its lock;
                # leave the observable state as STOPPING and let polling expose
                # the eventual STOPPED/FAILED result.
                raise SimulationStopPending(
                    f"模拟仍在停止中，图谱写入未在 {wait_timeout:.0f}s 内完成"
                )
        else:
            # Restart recovery or tests may have no monitor thread. Complete
            # the same barrier synchronously in this request.
            with cls._finalization_lock(simulation_id):
                state = cls.get_run_state(simulation_id) or state
                if cls._graph_memory_enabled.get(simulation_id, False):
                    try:
                        ZepGraphMemoryManager.stop_updater(simulation_id)
                        cls._graph_memory_enabled.pop(simulation_id, None)
                    except Exception as error:
                        state.runner_status = RunnerStatus.FAILED
                        state.twitter_running = False
                        state.reddit_running = False
                        state.completed_at = datetime.now().isoformat()
                        state.error = f"Zep图谱写入未完整完成: {error}"
                        cls._save_run_state(state)

View on GitHub (pinned to b5b53acc57)

Solutions

1. Do not treat this as failure: catch SimulationStopPending and keep polling get_run_state/status until runner_status becomes STOPPED or FAILED.
2. If it persists, check Zep API health/latency and network; raise ZEP_INGESTION_WAIT_TIMEOUT_SECONDS / ZEP_HTTP_REQUEST_TIMEOUT_SECONDS if your graphs are large.
3. Avoid stopping mid-write when possible — let the simulation finish naturally so the flush happens outside a stop deadline.

Example fix

// before
state = SimulationRunner.stop_simulation(sim_id);  # raises after ~35s of pending graph writes

// after
from app.services.simulation_runner import SimulationStopPending
try:
    state = SimulationRunner.stop_simulation(sim_id)
except SimulationStopPending:
    state = SimulationRunner.wait_terminal_state(sim_id, poll=5.0)  # poll runner_status until STOPPED/FAILED

from app.services.simulation_runner import SimulationStopPending

try:
    state = SimulationRunner.stop_simulation(sim_id)
except SimulationStopPending:
    state = poll_until_terminal(sim_id)  # GET status every 5s; STOPPING is expected here

When it happens

Trigger: Stopping a graph-memory-enabled simulation while the Zep updater is still flushing activity batches to the Zep Cloud API, with each HTTP request bounded by ZEP_HTTP_REQUEST_TIMEOUT; total flush exceeds the computed wait_timeout.

Common situations: Slow or rate-limited Zep Cloud API at stop time; large activity backlog (long simulations) still being batched; network degradation between backend and Zep.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Zep图谱写入未完整完成: {error}
• 启用图谱记忆更新时必须提供 graph_id
• Zep图谱更新器初始化失败: {e}
• Zep batch {submission.batch_id} did not finish within {timeo
• Zep episode processing timed out with {len(pending_episodes)

AI-assisted analysis of 666ghj/MiroFish@b5b53acc57 (2026-08-14). Data as JSON: /api/errors/d2048bf6ba4e56b1. Report an issue: GitHub.