BerriAI/litellm · error · SemanticToolFilterContextWindowError

MCP semantic tool filtering could not run: embedding model '

Error message

MCP semantic tool filtering could not run: embedding model '{embedding_model}' exceeded its context window while embedding {stage}. The request was blocked instead of silently passing all tools through. Switch to an embedding model with a larger context window, or disable semantic tool filtering.

What it means

With litellm_settings.mcp_semantic_tool_filter.enabled: true, LiteLLM builds a semantic index over all registered MCP tool descriptions. If that build fails because the embedding model's context window was exceeded, the failure is stored on the filter; every later filter call re-raises SemanticToolFilterContextWindowError instead of silently passing all tools through. This is a deliberate fail-closed design: the request is blocked rather than risking an oversized tool payload.

Source

Thrown at litellm/proxy/_experimental/mcp_server/semantic_tool_filter.py:285

        Filter tools semantically based on query.

        Args:
            query: User query to match against tools
            available_tools: Full list of available MCP tools
            top_k: Override default top_k (optional)

        Returns:
            Filtered and ordered list of tools (up to top_k)
        """
        # Early returns for cases where we can't/shouldn't filter
        if not self.enabled:
            return available_tools

        if not available_tools:
            return available_tools

        if self.context_window_error is not None:
            raise SemanticToolFilterContextWindowError(
                embedding_model=self.embedding_model,
                stage="the MCP tool descriptions during semantic router build",
                original_error=self.context_window_error,
            )

        if not query or not query.strip():
            return available_tools

        # Run semantic filtering
        try:
            await self._ensure_tools_indexed(available_tools)

            if self.tool_router is None:
                verbose_logger.warning("Semantic router could not be built from the request's tools")
                return available_tools

            available_names: Final = [name for name in (self._extract_tool_info(t)[0] for t in available_tools) if name]
            if not available_names:

View on GitHub (pinned to 77b7c6c40c)

Solutions

1. Configure a larger-context embedding model under litellm_settings.mcp_semantic_tool_filter.embedding_model (e.g., voyage-3 with a 32k window).
2. Reduce indexed content: unregister unused MCP servers or shorten their tool descriptions.
3. Disable the filter — litellm_settings.mcp_semantic_tool_filter.enabled: false — to fall back to passing tools through unfiltered (then watch the LLM's own context budget).
4. Restart/rebuild the semantic router after fixing either the model or the catalog.

Example fix

# before
litellm_settings:
  mcp_semantic_tool_filter:
    enabled: true
    # default embedding model context window too small for the catalog

# after
litellm_settings:
  mcp_semantic_tool_filter:
    enabled: true
    embedding_model: voyage/voyage-3

def approx_index_tokens(tools: list[dict]) -> int:
    return sum(len(t.get("description") or "") + len(t.get("name") or "") for t in tools) // 4

# before enabling the filter, make sure the catalog fits the embedding window
if approx_index_tokens(all_tools) > 7000:  # safety margin under an 8191-token window
    disable_semantic_filter_or_switch_model()

try:
    resp = await client.post(f"{base}/v1/chat/completions", json=payload)
    resp.raise_for_status()
except (httpx.HTTPStatusError, Exception) as e:
    if "exceeded its context window" in str(e) and "semantic tool filtering" in str(e):
        # fail-closed by design: fix config (bigger-window embedding model or
        # enabled: false), then replay the request; do not blind-retry
        raise ToolCatalogTooLarge(str(e)) from e
    raise

When it happens

Trigger: Many MCP servers registered so the concatenated tool descriptions exceed the embedding model's token limit (8191 for text-embedding-3-*); the startup build records the overflow in self.context_window_error, and each chat completion using MCP tools then raises this error from the early-return check.

Common situations: Onboarding dozens of MCP servers with verbose tool descriptions while keeping the default embedding model; catalogs that grew past the token ceiling after new servers were added.

Related errors

• {error_str}
• api_base is required for hosted_vllm embeddings
• OpenAPI spec not found at {filepath}
• error.summary
• ip_filtering

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/32d9380518443a3c. Report an issue: GitHub.