What causes "TogetherAIException - {error_response['error']}" in BerriAI/litellm?

Reduce max_tokens so prompt_tokens + max_tokens <= model context window

BerriAI/litellm · error · ContextWindowExceededError

TogetherAIException - {error_response['error']}

Error message

TogetherAIException - {error_response['error']}

What it means

This is a litellm ContextWindowExceededError raised by the Together AI exception mapper. The mapper first json.loads the error body; if it contains an 'error' key whose text includes '`inputs` tokens + `max_new_tokens` must be <=', Together AI has rejected the request because prompt tokens plus requested generation length exceed the model's context window. litellm normalizes this to its context-window class.

Source

Thrown at litellm/litellm_core_utils/exception_mapping_utils.py:1662

            )


def _map_together_ai_exception(
    *,
    model: str,
    original_exception: _ProviderHTTPException,
    custom_llm_provider: str,
    error_str: str,
    exception_type: str,
    exception_provider: str,
    extra_information: str,
) -> None:
    try:
        error_response = json.loads(error_str)
    except Exception:
        error_response = {"error": error_str}
    if "error" in error_response and "`inputs` tokens + `max_new_tokens` must be <=" in error_response["error"]:
        raise ContextWindowExceededError(
            message=f"TogetherAIException - {error_response['error']}",
            model=model,
            llm_provider="together_ai",
            response=getattr(original_exception, "response", None),
        )
    elif "error" in error_response and "invalid private key" in error_response["error"]:
        raise AuthenticationError(
            message=f"TogetherAIException - {error_response['error']}",
            llm_provider="together_ai",
            model=model,
            response=getattr(original_exception, "response", None),
        )
    elif "error" in error_response and "INVALID_ARGUMENT" in error_response["error"]:
        raise BadRequestError(
            message=f"TogetherAIException - {error_response['error']}",
            model=model,
            llm_provider="together_ai",
            response=getattr(original_exception, "response", None),

View on GitHub (pinned to 6c2dcb801b)

Solutions

Reduce max_tokens so prompt_tokens + max_tokens <= model context window
Truncate/summarize the prompt
Look up the model's limit: litellm.get_max_tokens(model)
Or switch to a larger-context Together model

Example fix

# before
litellm.completion(model="together_ai/Together-Code-7B", messages=msgs, max_tokens=2048)

# after
ctx = litellm.get_max_tokens("together_ai/Together-Code-7B")
prompt_toks = len(litellm.encode(model="together_ai/Together-Code-7B", text=msgs_text))
litellm.completion(..., max_tokens=min(512, ctx - prompt_toks - 8))

Defensive patterns

Strategy: validation

Validate before calling

ctx = litellm.get_max_tokens(model) or 4096
prompt_tokens = sum(
    len(litellm.encode(model=model, text=m["content"])) for m in messages
)
if prompt_tokens + max_tokens > ctx:
    max_tokens = max(64, ctx - prompt_tokens - 8)  # shrink output budget
    messages = trim_to(messages, ctx - max_tokens - 8)

Type guard

def is_context_window_error(e: Exception) -> bool:
    return isinstance(e, litellm.ContextWindowExceededError)

Try / catch

try:
    resp = litellm.completion(model=model, messages=messages, max_tokens=max_tokens)
except litellm.ContextWindowExceededError:
    resp = litellm.completion(model=model, messages=trim(messages, 0.5), max_tokens=256)

Prevention

Budget tokens: prompt_tokens + max_tokens must be < context window
Compute prompt token counts with litellm.encode before large calls
Don't set max_tokens reflexively high; size it to actual need

When it happens

Trigger: Calling a together_ai model with (prompt tokens + max_tokens) above the model context limit — e.g. long prompt with max_tokens=2048 on a 4096-context model; the check is on the SUM, not just the prompt.

Common situations: Developers setting generous max_tokens 'to be safe' which then overflows the budget; long RAG contexts; switching to a smaller-context model without adjusting max_tokens.

Related errors

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/cfb995e10efd88ed. Report an issue: GitHub.