BerriAI/litellm · error · ContextWindowExceededError

AI21Exception

Error message

AI21Exception - {original_exception.message}

What it means

LiteLLM's _map_ai21_exception raises ContextWindowExceededError when the AI21 error message contains 'Prompt has too many tokens'. It is LiteLLM's normalized signal that the input exceeded the model's maximum context length (e.g. Jurrassic-2 / Jamba limits).

Solutions

  1. Count and truncate inputs before the call: use litellm.token_counter(model, messages=messages) and trim history/chunks to fit.
  2. Switch to a larger-context AI21 model (e.g. jamba-1.5-large) or another provider via Router fallback.
  3. Implement sliding-window or summarize-older-turns trimming for chat history.
  4. Catch ContextWindowExceededError specifically and degrade gracefully (truncate + retry once).

Example fix

# before
litellm.completion(model='ai21/jamba-1.5-mini', messages=messages)  # too many tokens

# after
from litellm import token_counter
MAX = 100_000
while messages and token_counter(model='ai21/jamba-1.5-mini', messages=messages) > MAX:
    messages.pop(0)
litellm.completion(model='ai21/jamba-1.5-mini', messages=messages)
Defensive patterns

Strategy: validation

Validate before calling

from litellm import token_counter

def fits_context(model: str, messages: list, budget: int = 250_000) -> bool:
    try:
        return token_counter(model=model, messages=messages) < budget
    except Exception:
        return len(str(messages)) // 4 < budget  # rough fallback estimate

Type guard

from litellm import ContextWindowExceededError

def is_context_exceeded(e: BaseException) -> bool:
    return isinstance(e, ContextWindowExceededError)

Try / catch

from litellm import ContextWindowExceededError

try:
    resp = litellm.completion(model='ai21/jamba-1.5-mini', messages=messages)
except ContextWindowExceededError:
    messages = messages[: len(messages) // 2]  # trim oldest turns
    resp = litellm.completion(model='ai21/jamba-1.5-mini', messages=messages)

Prevention

When it happens

Trigger: completion('ai21/<model>', messages=long_history) where total tokens exceed the model's context window - AI21 rejects with 'Prompt has too many tokens' and this branch re-raises it as ContextWindowExceededError.

Common situations: Long chat histories without truncation, RAG pipelines stuffing too many retrieved chunks, or switching from a large-context model (Jamba 256k) to a smaller-context one without re-tuning prompt budgets.

Related errors

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/48ecfeb39a953ec7. Report an issue: GitHub.

Appendix: source

Thrown at litellm/litellm_core_utils/exception_mapping_utils.py:1505

                llm_provider="huggingface",
                model=model,
                request=getattr(original_exception, "request", None),
            )


def _map_ai21_exception(
    *,
    model: str,
    original_exception: _ProviderHTTPException,
    custom_llm_provider: str,
    error_str: str,
    exception_type: str,
    exception_provider: str,
    extra_information: str,
) -> None:
    if hasattr(original_exception, "message"):
        if "Prompt has too many tokens" in original_exception.message:
            raise ContextWindowExceededError(
                message=f"AI21Exception - {original_exception.message}",
                model=model,
                llm_provider="ai21",
                response=getattr(original_exception, "response", None),
            )
        if "Bad or missing API token." in original_exception.message:
            raise BadRequestError(
                message=f"AI21Exception - {original_exception.message}",
                model=model,
                llm_provider="ai21",
                response=getattr(original_exception, "response", None),
            )
    if hasattr(original_exception, "status_code"):
        if original_exception.status_code == 401:
            raise AuthenticationError(
                message=f"AI21Exception - {original_exception.message}",
                llm_provider="ai21",
                model=model,

View on GitHub (pinned to 77b7c6c40c)