BerriAI/litellm · error · ContextWindowExceededError

TogetherAIException

Error message

TogetherAIException - {error_response['error']}

What it means

Together AI rejects a request whose prompt tokens plus max_new_tokens exceed the model's context window; litellm detects the '`inputs` tokens + `max_new_tokens` must be <=' substring in the parsed JSON error body and re-raises it as litellm.ContextWindowExceededError (a BadRequestError subclass). The Together error message names the actual limit, so read it to see the ceiling you crossed.

Solutions

1. Lower max_tokens so prompt tokens + max_new_tokens fit the limit stated in the error message
2. Trim, chunk, or summarize the prompt; drop few-shot examples
3. Pre-flight the budget: litellm.token_counter(model, messages) vs litellm.get_max_tokens(model)
4. Switch to a long-context Together model variant when the prompt legitimately needs it

Example fix

# before
litellm.completion(model='together_ai/meta-llama/Llama-3-8b-chat-hf', messages=docs, max_tokens=4096)

# after: clamp the generation budget to what is left
limit = litellm.get_max_tokens('together_ai/meta-llama/Llama-3-8b-chat-hf')
used = litellm.token_counter(model='together_ai/meta-llama/Llama-3-8b-chat-hf', messages=docs)
litellm.completion(model='together_ai/meta-llama/Llama-3-8b-chat-hf', messages=docs, max_tokens=min(4096, limit - used - 8))

model = 'together_ai/meta-llama/Llama-3-8b-chat-hf'
limit = litellm.get_max_tokens(model)
used = litellm.token_counter(model=model, messages=msgs)
if used + requested_max_tokens > limit:
    requested_max_tokens = max(64, limit - used - 8)  # clamp to fit

def is_context_window_error(e: BaseException) -> bool:
    return isinstance(e, litellm.ContextWindowExceededError)

try:
    resp = litellm.completion(model=model, messages=msgs, max_tokens=mt)
except litellm.ContextWindowExceededError:
    msgs = truncate_to_budget(msgs, limit - mt - 8)
    resp = litellm.completion(model=model, messages=msgs, max_tokens=mt)

When it happens

Trigger: chat/completion calls on model='together_ai/<model>' where token_counter(prompt) + max_tokens > context limit (e.g. 8192 on base models): long few-shot prompts, whole documents pasted in, or a large default max_tokens combined with a medium prompt.

Common situations: RAG pipelines stuffing retrieved chunks untrimmed; a global max_tokens=4096 default reused across models; downgrading from a 32k-context model to a standard one without resizing prompts.

Related errors

• AI21Exception
• AlephAlphaException
• ContextWindowExceededError
• Exception: Context Window Error
• Exception

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/cfb995e10efd88ed. Report an issue: GitHub.

Appendix: source

Thrown at litellm/litellm_core_utils/exception_mapping_utils.py:1662

            )


def _map_together_ai_exception(
    *,
    model: str,
    original_exception: _ProviderHTTPException,
    custom_llm_provider: str,
    error_str: str,
    exception_type: str,
    exception_provider: str,
    extra_information: str,
) -> None:
    try:
        error_response = json.loads(error_str)
    except Exception:
        error_response = {"error": error_str}
    if "error" in error_response and "`inputs` tokens + `max_new_tokens` must be <=" in error_response["error"]:
        raise ContextWindowExceededError(
            message=f"TogetherAIException - {error_response['error']}",
            model=model,
            llm_provider="together_ai",
            response=getattr(original_exception, "response", None),
        )
    elif "error" in error_response and "invalid private key" in error_response["error"]:
        raise AuthenticationError(
            message=f"TogetherAIException - {error_response['error']}",
            llm_provider="together_ai",
            model=model,
            response=getattr(original_exception, "response", None),
        )
    elif "error" in error_response and "INVALID_ARGUMENT" in error_response["error"]:
        raise BadRequestError(
            message=f"TogetherAIException - {error_response['error']}",
            model=model,
            llm_provider="together_ai",
            response=getattr(original_exception, "response", None),

View on GitHub (pinned to 77b7c6c40c)