HKUDS/DeepTutor · error · IndexingStallError

Indexing made no progress for {stalled_for:.0f}s while embed

Error message

Indexing made no progress for {stalled_for:.0f}s while embedding documents. The embedding provider may be accepting requests without completing them; check the embedding endpoint and retry.

What it means

Raised by the RAG pipeline's stall guard when the embedding job makes no reported progress for longer than the stall timeout while embedding documents. It indicates the embedding provider is accepting requests but not completing them (hung or black-holed HTTP calls). The guard reclaims the shared progress callback slot before raising so concurrent jobs don't mask the failure.

Source

Thrown at deeptutor/services/rag/pipelines/llamaindex/pipeline.py:107

    set_progress_callback(_heartbeat)
    future = asyncio.get_running_loop().run_in_executor(None, fn)

    def _consume_terminal_exception(fut: "asyncio.Future[Any]") -> None:
        # The stalled thread may finish after we raise; retrieve its exception
        # so it is not reported as "exception was never retrieved".
        if not fut.cancelled():
            fut.exception()

    while True:
        done, _ = await asyncio.wait({future}, timeout=_INDEX_STALL_POLL_SECONDS)
        if done:
            return future.result()
        # Reclaim the shared callback slot in case a concurrent job took it.
        set_progress_callback(_heartbeat)
        stalled_for = time.monotonic() - last_progress["at"]
        if stalled_for > stall_timeout:
            future.add_done_callback(_consume_terminal_exception)
            raise IndexingStallError(
                f"Indexing made no progress for {stalled_for:.0f}s while "
                "embedding documents. The embedding provider may be accepting "
                "requests without completing them; check the embedding "
                "endpoint and retry."
            )


class LlamaIndexPipeline:
    """Pipeline that indexes and retrieves KB content via LlamaIndex."""

    def __init__(
        self,
        kb_base_dir: Optional[str] = None,
        *,
        signature_provider: SignatureProvider | None = None,
        document_loader: LlamaIndexDocumentLoader | None = None,
    ):
        self.logger = logging.getLogger(__name__)

View on GitHub (pinned to 3e82f13042)

Solutions

Check the embedding endpoint health: curl the provider's /embeddings route with a tiny payload and confirm a timely 200.
Verify embedding provider config (base_url, API key, model name) in runtime settings; a wrong model can hang some servers.
Retry after lowering batch size or increasing stall_timeout if the provider is legitimately slow.
If using a local embedding server, restart it and confirm it logs each request; then retry add_documents.

Example fix

# before
index = pipeline.initialize()  # hangs then raises IndexingStallError after stall_timeout

# after
import time
# verify provider responds quickly before indexing
assert embedding_client.ping() < 5.0, "embedding endpoint unresponsive"
try:
    index = pipeline.initialize()
except IndexingStallError:
    logger.error("embedding provider stalled; check endpoint and retry")
    raise

Defensive patterns

Strategy: retry

Validate before calling

import time
start = time.monotonic()
probe = embedding_client.embed(["ping"])
assert time.monotonic() - start < 10, "embedding endpoint too slow/unresponsive"

Try / catch

try:
    pipeline.add_documents(docs, progress_callback=cb)
except IndexingStallError as e:
    logger.warning("stall: %s", e)
    time.sleep(backoff)
    pipeline.add_documents(docs)  # one bounded retry

Prevention

Health-check the embedding endpoint before large indexing runs.
Keep a progress callback wired so the stall guard sees heartbeats.
Set stall_timeout relative to realistic batch latency; monitor provider status pages.
Run big ingests in smaller batches so a stall is cheap to retry.

When it happens

Trigger: Calling initialize() or add_documents() on the LlamaIndex RAG pipeline where the embedding endpoint accepts connections but never returns (e.g., proxy stalls, provider outage, rate-limited streaming), so last_progress timestamp never advances past stall_timeout.

Common situations: Misconfigured embedding base_url pointing to a dead endpoint, an OpenAI-compatible server that queues requests indefinitely, network/proxy interruptions mid-batch, or an embedding provider under severe throttling.

Related errors

AI-assisted analysis of HKUDS/DeepTutor@3e82f13042 (2026-08-27). Data as JSON: /api/errors/d3d2d0a854172bf8. Report an issue: GitHub.