{"record":{"id":"d3d2d0a854172bf8","repo":"HKUDS/DeepTutor","slug":"indexing-made-no-progress-for-stalled-for-0f-s-w","errorCode":null,"errorMessage":"Indexing made no progress for {stalled_for:.0f}s while embedding documents. The embedding provider may be accepting requests without completing them; check the embedding endpoint and retry.","messagePattern":"Indexing made no progress for (.+?)s while embedding documents\\. The embedding provider may be accepting requests without completing them; check the embedding endpoint and retry\\.","errorType":"exception","errorClass":"IndexingStallError","httpStatus":null,"severity":"error","filePath":"deeptutor/services/rag/pipelines/llamaindex/pipeline.py","lineNumber":107,"sourceCode":"    set_progress_callback(_heartbeat)\n    future = asyncio.get_running_loop().run_in_executor(None, fn)\n\n    def _consume_terminal_exception(fut: \"asyncio.Future[Any]\") -> None:\n        # The stalled thread may finish after we raise; retrieve its exception\n        # so it is not reported as \"exception was never retrieved\".\n        if not fut.cancelled():\n            fut.exception()\n\n    while True:\n        done, _ = await asyncio.wait({future}, timeout=_INDEX_STALL_POLL_SECONDS)\n        if done:\n            return future.result()\n        # Reclaim the shared callback slot in case a concurrent job took it.\n        set_progress_callback(_heartbeat)\n        stalled_for = time.monotonic() - last_progress[\"at\"]\n        if stalled_for > stall_timeout:\n            future.add_done_callback(_consume_terminal_exception)\n            raise IndexingStallError(\n                f\"Indexing made no progress for {stalled_for:.0f}s while \"\n                \"embedding documents. The embedding provider may be accepting \"\n                \"requests without completing them; check the embedding \"\n                \"endpoint and retry.\"\n            )\n\n\nclass LlamaIndexPipeline:\n    \"\"\"Pipeline that indexes and retrieves KB content via LlamaIndex.\"\"\"\n\n    def __init__(\n        self,\n        kb_base_dir: Optional[str] = None,\n        *,\n        signature_provider: SignatureProvider | None = None,\n        document_loader: LlamaIndexDocumentLoader | None = None,\n    ):\n        self.logger = logging.getLogger(__name__)","sourceCodeStart":89,"sourceCodeEnd":125,"githubUrl":"https://github.com/HKUDS/DeepTutor/blob/3e82f130422a813cdd73c10b21a44e9325f5821a/deeptutor/services/rag/pipelines/llamaindex/pipeline.py#L89-L125","documentation":"Raised by the RAG pipeline's stall guard when the embedding job makes no reported progress for longer than the stall timeout while embedding documents. It indicates the embedding provider is accepting requests but not completing them (hung or black-holed HTTP calls). The guard reclaims the shared progress callback slot before raising so concurrent jobs don't mask the failure.","triggerScenarios":"Calling initialize() or add_documents() on the LlamaIndex RAG pipeline where the embedding endpoint accepts connections but never returns (e.g., proxy stalls, provider outage, rate-limited streaming), so last_progress timestamp never advances past stall_timeout.","commonSituations":"Misconfigured embedding base_url pointing to a dead endpoint, an OpenAI-compatible server that queues requests indefinitely, network/proxy interruptions mid-batch, or an embedding provider under severe throttling.","solutions":["Check the embedding endpoint health: curl the provider's /embeddings route with a tiny payload and confirm a timely 200.","Verify embedding provider config (base_url, API key, model name) in runtime settings; a wrong model can hang some servers.","Retry after lowering batch size or increasing stall_timeout if the provider is legitimately slow.","If using a local embedding server, restart it and confirm it logs each request; then retry add_documents."],"exampleFix":"# before\nindex = pipeline.initialize()  # hangs then raises IndexingStallError after stall_timeout\n\n# after\nimport time\n# verify provider responds quickly before indexing\nassert embedding_client.ping() < 5.0, \"embedding endpoint unresponsive\"\ntry:\n    index = pipeline.initialize()\nexcept IndexingStallError:\n    logger.error(\"embedding provider stalled; check endpoint and retry\")\n    raise","handlingStrategy":"retry","validationCode":"import time\nstart = time.monotonic()\nprobe = embedding_client.embed([\"ping\"])\nassert time.monotonic() - start < 10, \"embedding endpoint too slow/unresponsive\"","typeGuard":null,"tryCatchPattern":"try:\n    pipeline.add_documents(docs, progress_callback=cb)\nexcept IndexingStallError as e:\n    logger.warning(\"stall: %s\", e)\n    time.sleep(backoff)\n    pipeline.add_documents(docs)  # one bounded retry","preventionTips":["Health-check the embedding endpoint before large indexing runs.","Keep a progress callback wired so the stall guard sees heartbeats.","Set stall_timeout relative to realistic batch latency; monitor provider status pages.","Run big ingests in smaller batches so a stall is cheap to retry."],"tags":["rag","embedding","timeout","stall-detection","llamaindex"],"backgroundTag":"request-timeout-stall","analyzedSha":"3e82f130422a813cdd73c10b21a44e9325f5821a","analyzedAt":"2026-08-27T06:57:25.364Z","schemaVersion":2},"datasetVersion":"2026-08-27T08:17:20.692Z"}