BerriAI/litellm · error · Timeout

TogetherAIException - {original_exception.message}

Error message

TogetherAIException - {original_exception.message}

What it means

litellm translates an HTTP 408 from the Together AI API into litellm.Timeout ('TogetherAIException - ...'). It means Together's server accepted the request but it did not complete within its gateway timeout window. litellm re-raises it as a Timeout so callers can apply uniform retry logic.

Source

Thrown at litellm/litellm_core_utils/exception_mapping_utils.py:1702

            message=f"TogetherAIException - {error_str}",
            model=model,
            llm_provider="together_ai",
        )
    elif (
        "error" in error_response
        and "API key doesn't match expected format." in error_response["error"]
        or "error_type" in error_response
        and error_response["error_type"] == "validation"
    ):
        raise BadRequestError(
            message=f"TogetherAIException - {error_response['error']}",
            model=model,
            llm_provider="together_ai",
            response=getattr(original_exception, "response", None),
        )
    if hasattr(original_exception, "status_code"):
        if original_exception.status_code == 408:
            raise Timeout(
                message=f"TogetherAIException - {original_exception.message}",
                model=model,
                llm_provider="together_ai",
            )
        elif original_exception.status_code == 422:
            raise BadRequestError(
                message=f"TogetherAIException - {error_response['error']}",
                model=model,
                llm_provider="together_ai",
                response=getattr(original_exception, "response", None),
            )
        elif original_exception.status_code == 429:
            raise RateLimitError(
                message=f"TogetherAIException - {original_exception.message}",
                llm_provider="together_ai",
                model=model,
                response=getattr(original_exception, "response", None),
            )

View on GitHub (pinned to 6c2dcb801b)

Solutions

1. Retry with exponential backoff (408 is transient) using litellm.Timeout as the catch signal
2. Reduce max_tokens, trim the prompt, or set stream=True so tokens flow before the gateway timeout
3. Pass an explicit timeout to litellm.completion(...) so the client gives up before the server does
4. If persistent, check Together AI status/usage dashboard for degraded throughput

Example fix

// before
resp = litellm.completion(model='together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1', messages=msgs, max_tokens=4096)
// after
resp = litellm.completion(model='together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1', messages=msgs, max_tokens=1024, stream=True, timeout=120)

prompt_tokens = litellm.token_counter(model='together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1', messages=msgs)
if prompt_tokens > 20_000:  # long prompts push past gateway timeout
    raise ValueError('Trim prompt or stream before calling Together AI')

import litellm

def is_together_timeout(e: Exception) -> bool:
    return isinstance(e, litellm.Timeout) and 'TogetherAIException' in str(e)

for attempt in range(3):
    try:
        return litellm.completion(...)
    except litellm.Timeout:
        time.sleep(2 ** attempt)
raise RuntimeError('Together AI timed out after retries')

When it happens

Trigger: A together_ai completion/embedding call whose server-side processing exceeds Together's request timeout, so the API returns status 408. Typical with very large prompts, huge max_tokens, or slow non-streaming generations.

Common situations: Long-context jobs on Mixtral/Llama models routed via model='together_ai/...'; sudden latency spikes on Together's side; requests serialized through slow middlewares that inflate processing time.

Related errors

• TogetherAIException - {error_str}
• Task {task_id} did not complete within {max_attempts * poll_
• BedrockException: Rate Limit Error - {error_str}
• BedrockException: Timeout Error - {error_str}
• TogetherAIException - {error_response['error']}

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/d5ca49ce0c7725c6. Report an issue: GitHub.