{"record":{"id":"03915fcef81dc040","repo":"666ghj/MiroFish","slug":"zep-batch-submission-batch-id-did-not-finish-wit","errorCode":null,"errorMessage":"Zep batch {submission.batch_id} did not finish within {timeout}s","messagePattern":"Zep batch (.+?) did not finish within (.+?)s","errorType":"exception","errorClass":"TimeoutError","httpStatus":null,"severity":"error","filePath":"backend/app/services/graph_builder.py","lineNumber":645,"sourceCode":"            lambda: self.client.batch.get(batch_id=batch_id),\n            operation_name=f\"get batch {batch_id}\",\n        )\n\n    def _wait_for_batch(\n        self,\n        submission: BatchSubmission,\n        progress_callback: Optional[Callable] = None,\n        timeout: int | None = None,\n    ) -> List[str]:\n        \"\"\"Wait for a Batch API terminal state and validate every item.\"\"\"\n\n        timeout = timeout or ZEP_INGESTION_WAIT_TIMEOUT_SECONDS\n        start_time = time.time()\n        terminal_states = {\"succeeded\", \"partial\", \"failed\", \"invalid\", \"canceled\"}\n\n        while True:\n            if time.time() - start_time > timeout:\n                raise TimeoutError(\n                    f\"Zep batch {submission.batch_id} did not finish within {timeout}s\"\n                )\n\n            summary = call_zep_read_with_retry(\n                lambda: self.client.batch.get(batch_id=submission.batch_id),\n                operation_name=f\"poll batch {submission.batch_id}\",\n            )\n            status = getattr(summary, \"status\", None)\n            progress = getattr(summary, \"progress\", None)\n            percent = float(getattr(progress, \"percent_complete\", 0) or 0) / 100\n            if progress_callback:\n                completed = int(getattr(progress, \"succeeded_items\", 0) or 0)\n                progress_callback(\n                    t(\n                        'progress.zepProcessing',\n                        completed=completed,\n                        total=submission.item_count,\n                        pending=max(submission.item_count - completed, 0),","sourceCodeStart":627,"sourceCodeEnd":663,"githubUrl":"https://github.com/666ghj/MiroFish/blob/b5b53acc57189a4a42e44a23e149dc655c98fe82/backend/app/services/graph_builder.py#L627-L663","documentation":"Raised by GraphBuilder._wait_for_batch when polling client.batch.get(batch_id=...) does not reach a terminal state (succeeded/partial/failed/invalid/canceled) within the timeout. The default timeout is ZEP_INGESTION_WAIT_TIMEOUT_SECONDS = 600 seconds (backend/app/utils/zep.py:28). It is a TimeoutError raised client-side; the batch may still complete on Zep's side later.","triggerScenarios":"Submitting a large batch (up to 50,000 items) whose server-side processing exceeds the wait window; polling loop spaced by time.sleep(3) while Zep's queue is backlogged; passing an explicit small timeout parameter; or a batch stuck in a non-terminal state on Zep's side.","commonSituations":"Ingesting a big document set during peak load; Zep cloud slowness or incidents; a tight custom timeout passed by a test; blocking a web request handler for 10 minutes until this fires.","solutions":["Raise ZEP_INGESTION_WAIT_TIMEOUT_SECONDS (or the per-call timeout argument) to comfortably exceed your worst-case batch processing time.","Split very large submissions into several smaller batches so each finishes within the window.","After the timeout, poll the batch once more manually via client.batch.get before declaring failure — it often finishes shortly after.","Restructure to an async/job model so the wait does not hold a request thread hostage for 600s."],"exampleFix":"# before\nresult = builder._wait_for_batch(submission)  # default 600s\n# after\nresult = builder._wait_for_batch(\n    submission,\n    timeout=max(600, submission.item_count // 10),  # scale with batch size\n)","handlingStrategy":"retry","validationCode":"timeout = max(ZEP_INGESTION_WAIT_TIMEOUT_SECONDS, item_count * 2)  # scale wait with batch size","typeGuard":null,"tryCatchPattern":"try:\n    result = builder._wait_for_batch(submission, timeout=scaled_timeout)\nexcept TimeoutError:\n    summary = client.batch.get(batch_id=submission.batch_id)\n    if getattr(summary, 'status', None) == 'succeeded':\n        result = builder._list_batch_items(submission.batch_id)\n    else:\n        raise","preventionTips":["Size batches so worst-case processing fits the wait window.","Scale the timeout with item_count instead of using a flat 600s.","Move batch waits into background jobs so timeouts don't block request threads."],"tags":["zep","batch-api","timeout","polling","ingestion"],"backgroundTag":null,"analyzedSha":"b5b53acc57189a4a42e44a23e149dc655c98fe82","analyzedAt":"2026-08-14T22:29:33.146Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}