{"record":{"id":"cfb995e10efd88ed","repo":"BerriAI/litellm","slug":"togetheraiexception-error-response-error","errorCode":null,"errorMessage":"TogetherAIException - {error_response['error']}","messagePattern":"TogetherAIException - (.+?)","errorType":"exception","errorClass":"ContextWindowExceededError","httpStatus":400,"severity":"error","filePath":"litellm/litellm_core_utils/exception_mapping_utils.py","lineNumber":1662,"sourceCode":"            )\n\n\ndef _map_together_ai_exception(\n    *,\n    model: str,\n    original_exception: _ProviderHTTPException,\n    custom_llm_provider: str,\n    error_str: str,\n    exception_type: str,\n    exception_provider: str,\n    extra_information: str,\n) -> None:\n    try:\n        error_response = json.loads(error_str)\n    except Exception:\n        error_response = {\"error\": error_str}\n    if \"error\" in error_response and \"`inputs` tokens + `max_new_tokens` must be <=\" in error_response[\"error\"]:\n        raise ContextWindowExceededError(\n            message=f\"TogetherAIException - {error_response['error']}\",\n            model=model,\n            llm_provider=\"together_ai\",\n            response=getattr(original_exception, \"response\", None),\n        )\n    elif \"error\" in error_response and \"invalid private key\" in error_response[\"error\"]:\n        raise AuthenticationError(\n            message=f\"TogetherAIException - {error_response['error']}\",\n            llm_provider=\"together_ai\",\n            model=model,\n            response=getattr(original_exception, \"response\", None),\n        )\n    elif \"error\" in error_response and \"INVALID_ARGUMENT\" in error_response[\"error\"]:\n        raise BadRequestError(\n            message=f\"TogetherAIException - {error_response['error']}\",\n            model=model,\n            llm_provider=\"together_ai\",\n            response=getattr(original_exception, \"response\", None),","sourceCodeStart":1644,"sourceCodeEnd":1680,"githubUrl":"https://github.com/BerriAI/litellm/blob/77b7c6c40c0c5aa5fbcb1d6a1825ac39ca8829b8/litellm/litellm_core_utils/exception_mapping_utils.py#L1644-L1680","documentation":"Together AI rejects a request whose prompt tokens plus max_new_tokens exceed the model's context window; litellm detects the '`inputs` tokens + `max_new_tokens` must be <=' substring in the parsed JSON error body and re-raises it as litellm.ContextWindowExceededError (a BadRequestError subclass). The Together error message names the actual limit, so read it to see the ceiling you crossed.","triggerScenarios":"chat/completion calls on model='together_ai/<model>' where token_counter(prompt) + max_tokens > context limit (e.g. 8192 on base models): long few-shot prompts, whole documents pasted in, or a large default max_tokens combined with a medium prompt.","commonSituations":"RAG pipelines stuffing retrieved chunks untrimmed; a global max_tokens=4096 default reused across models; downgrading from a 32k-context model to a standard one without resizing prompts.","solutions":["Lower max_tokens so prompt tokens + max_new_tokens fit the limit stated in the error message","Trim, chunk, or summarize the prompt; drop few-shot examples","Pre-flight the budget: litellm.token_counter(model, messages) vs litellm.get_max_tokens(model)","Switch to a long-context Together model variant when the prompt legitimately needs it"],"exampleFix":"# before\nlitellm.completion(model='together_ai/meta-llama/Llama-3-8b-chat-hf', messages=docs, max_tokens=4096)\n\n# after: clamp the generation budget to what is left\nlimit = litellm.get_max_tokens('together_ai/meta-llama/Llama-3-8b-chat-hf')\nused = litellm.token_counter(model='together_ai/meta-llama/Llama-3-8b-chat-hf', messages=docs)\nlitellm.completion(model='together_ai/meta-llama/Llama-3-8b-chat-hf', messages=docs, max_tokens=min(4096, limit - used - 8))","handlingStrategy":"validation","validationCode":"model = 'together_ai/meta-llama/Llama-3-8b-chat-hf'\nlimit = litellm.get_max_tokens(model)\nused = litellm.token_counter(model=model, messages=msgs)\nif used + requested_max_tokens > limit:\n    requested_max_tokens = max(64, limit - used - 8)  # clamp to fit","typeGuard":"def is_context_window_error(e: BaseException) -> bool:\n    return isinstance(e, litellm.ContextWindowExceededError)","tryCatchPattern":"try:\n    resp = litellm.completion(model=model, messages=msgs, max_tokens=mt)\nexcept litellm.ContextWindowExceededError:\n    msgs = truncate_to_budget(msgs, limit - mt - 8)\n    resp = litellm.completion(model=model, messages=msgs, max_tokens=mt)","preventionTips":["Pre-count tokens with litellm.token_counter and clamp max_tokens before every long call","Keep prompts chunked at ingestion time so no single call can exceed the window","Cache get_max_tokens per model instead of assuming 8192 everywhere"],"tags":["together-ai","litellm","context-window","max-tokens","token-limit"],"backgroundTag":"context-window-exceeded","analyzedSha":"77b7c6c40c0c5aa5fbcb1d6a1825ac39ca8829b8","analyzedAt":"2026-08-18T11:44:31.656Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}