666ghj/MiroFish · error · TimeoutError

Zep batch {submission.batch_id} did not finish within {timeo

Error message

Zep batch {submission.batch_id} did not finish within {timeout}s

What it means

Raised by GraphBuilder._wait_for_batch when polling client.batch.get(batch_id=...) does not reach a terminal state (succeeded/partial/failed/invalid/canceled) within the timeout. The default timeout is ZEP_INGESTION_WAIT_TIMEOUT_SECONDS = 600 seconds (backend/app/utils/zep.py:28). It is a TimeoutError raised client-side; the batch may still complete on Zep's side later.

Source

Thrown at backend/app/services/graph_builder.py:645

            lambda: self.client.batch.get(batch_id=batch_id),
            operation_name=f"get batch {batch_id}",
        )

    def _wait_for_batch(
        self,
        submission: BatchSubmission,
        progress_callback: Optional[Callable] = None,
        timeout: int | None = None,
    ) -> List[str]:
        """Wait for a Batch API terminal state and validate every item."""

        timeout = timeout or ZEP_INGESTION_WAIT_TIMEOUT_SECONDS
        start_time = time.time()
        terminal_states = {"succeeded", "partial", "failed", "invalid", "canceled"}

        while True:
            if time.time() - start_time > timeout:
                raise TimeoutError(
                    f"Zep batch {submission.batch_id} did not finish within {timeout}s"
                )

            summary = call_zep_read_with_retry(
                lambda: self.client.batch.get(batch_id=submission.batch_id),
                operation_name=f"poll batch {submission.batch_id}",
            )
            status = getattr(summary, "status", None)
            progress = getattr(summary, "progress", None)
            percent = float(getattr(progress, "percent_complete", 0) or 0) / 100
            if progress_callback:
                completed = int(getattr(progress, "succeeded_items", 0) or 0)
                progress_callback(
                    t(
                        'progress.zepProcessing',
                        completed=completed,
                        total=submission.item_count,
                        pending=max(submission.item_count - completed, 0),

View on GitHub (pinned to b5b53acc57)

Solutions

Raise ZEP_INGESTION_WAIT_TIMEOUT_SECONDS (or the per-call timeout argument) to comfortably exceed your worst-case batch processing time.
Split very large submissions into several smaller batches so each finishes within the window.
After the timeout, poll the batch once more manually via client.batch.get before declaring failure — it often finishes shortly after.
Restructure to an async/job model so the wait does not hold a request thread hostage for 600s.

Example fix

# before
result = builder._wait_for_batch(submission)  # default 600s
# after
result = builder._wait_for_batch(
    submission,
    timeout=max(600, submission.item_count // 10),  # scale with batch size
)

Defensive patterns

Strategy: retry

Validate before calling

timeout = max(ZEP_INGESTION_WAIT_TIMEOUT_SECONDS, item_count * 2)  # scale wait with batch size

Try / catch

try:
    result = builder._wait_for_batch(submission, timeout=scaled_timeout)
except TimeoutError:
    summary = client.batch.get(batch_id=submission.batch_id)
    if getattr(summary, 'status', None) == 'succeeded':
        result = builder._list_batch_items(submission.batch_id)
    else:
        raise

Prevention

Size batches so worst-case processing fits the wait window.
Scale the timeout with item_count instead of using a flat 600s.
Move batch waits into background jobs so timeouts don't block request threads.

When it happens

Trigger: Submitting a large batch (up to 50,000 items) whose server-side processing exceeds the wait window; polling loop spaced by time.sleep(3) while Zep's queue is backlogged; passing an explicit small timeout parameter; or a batch stuck in a non-terminal state on Zep's side.

Common situations: Ingesting a big document set during peak load; Zep cloud slowness or incidents; a tight custom timeout passed by a test; blocking a web request handler for 10 minutes until this fires.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of 666ghj/MiroFish@b5b53acc57 (2026-08-14). Data as JSON: /api/errors/03915fcef81dc040. Report an issue: GitHub.