BerriAI/litellm · error · ContextWindowExceededError

{error_str}

Error message

{error_str}

What it means

_map_huggingface_exception raises litellm.ContextWindowExceededError with the raw error string when a HuggingFace Inference Endpoints error contains 'length limit exceeded'. The model behind the endpoint rejected the prompt because it exceeds the model's max sequence length (often far smaller than commercial APIs — 4k/8k/32k depending on the deployed model). litellm maps it so context-truncation and fallback strategies can key off the exception type.

Solutions

1. Chunk inputs to the model's actual limit (check the model card for max sequence length, e.g. 512 for many embedders).
2. Trim chat history / summarize older turns before sending.
3. Switch to a long-context model on the endpoint (e.g. a 32k model) or a commercial provider via litellm fallback.
4. Pre-count tokens client-side with the model's tokenizer to fail before spending a request.

Example fix

# before
resp = litellm.embedding(model="huggingface/BAAI/bge-large-en-v1.5", input=[long_document_text])  # ContextWindowExceededError: length limit exceeded

# after
chunks = [long_document_text[i:i+1500] for i in range(0, len(long_document_text), 1500)]  # ~<512 tokens each
resp = litellm.embedding(model="huggingface/BAAI/bge-large-en-v1.5", input=chunks)

HF_SEQ_LIMITS = {"BAAI/bge-large-en-v1.5": 512, "sentence-transformers/all-MiniLM-L6-v2": 512}  # per model cards

def chunk_for_hf(text: str, model: str, chars_per_token: float = 4.0) -> list[str]:
    limit = HF_SEQ_LIMITS.get(model.split("/", 1)[-1], 512)
    max_chars = int(limit * chars_per_token * 0.9)
    return [text[i:i + max_chars] for i in range(0, len(text), max_chars)] or [""]

import litellm

def is_context_window_exceeded(e: BaseException) -> bool:
    return isinstance(e, litellm.ContextWindowExceededError)

try:
    resp = litellm.embedding(model="huggingface/BAAI/bge-large-en-v1.5", input=[doc])
except litellm.ContextWindowExceededError as e:
    logger.warning("doc over HF seq limit, chunking")
    resp = litellm.embedding(model="huggingface/BAAI/bge-large-en-v1.5", input=chunk_for_hf(doc, "BAAI/bge-large-en-v1.5"))

When it happens

Trigger: Sending prompts longer than the deployed model's max_seq_len to model='huggingface/<model>' via inference endpoints/embeddings; RAG contexts overflowing small models (e.g. bert-based embedders with 512-token limits); chat histories accumulated without trimming.

Common situations: Default embedding models with 512-token windows receiving whole paragraphs per chunk, text-generation endpoints running 8k models fed 20k-token transcripts, developers assuming commercial-API context sizes (128k) apply to HuggingFace deployments.

Related errors

• Invalid task_type= . Expected one of=
• reranker requires 2+ sentences
• sentence-similarity requires 2+ sentences
• AI21Exception
• AlephAlphaException

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/8e2c2d32565c536b. Report an issue: GitHub.

Appendix: source

Thrown at litellm/litellm_core_utils/exception_mapping_utils.py:1428

                llm_provider="cohere",
                model=model,
                request=getattr(original_exception, "request", None),
            )
        raise cast(Exception, original_exception)


def _map_huggingface_exception(
    *,
    model: str,
    original_exception: _ProviderHTTPException,
    custom_llm_provider: str,
    error_str: str,
    exception_type: str,
    exception_provider: str,
    extra_information: str,
) -> None:
    if "length limit exceeded" in error_str:
        raise ContextWindowExceededError(
            message=error_str,
            model=model,
            llm_provider="huggingface",
            response=getattr(original_exception, "response", None),
        )
    elif "A valid user token is required" in error_str:
        raise BadRequestError(
            message=error_str,
            llm_provider="huggingface",
            model=model,
            response=getattr(original_exception, "response", None),
        )
    elif "Rate limit reached" in error_str:
        raise RateLimitError(
            message=error_str,
            llm_provider="huggingface",
            model=model,
            response=getattr(original_exception, "response", None),

View on GitHub (pinned to 77b7c6c40c)