BerriAI/litellm · warning · Timeout

TogetherAIException - {error_str}

Error message

TogetherAIException - {error_str}

What it means

This is a litellm Timeout raised by the Together AI exception mapper when the raw error string contains 'A timeout occurred'. Together AI returns this textual message when the request exceeds its processing time limit; litellm matches it by substring on error_str (the raw body, not the parsed JSON) and raises the retryable Timeout class with the 'TogetherAIException - ' prefix.

Source

Thrown at litellm/litellm_core_utils/exception_mapping_utils.py:1683

            llm_provider="together_ai",
            response=getattr(original_exception, "response", None),
        )
    elif "error" in error_response and "invalid private key" in error_response["error"]:
        raise AuthenticationError(
            message=f"TogetherAIException - {error_response['error']}",
            llm_provider="together_ai",
            model=model,
            response=getattr(original_exception, "response", None),
        )
    elif "error" in error_response and "INVALID_ARGUMENT" in error_response["error"]:
        raise BadRequestError(
            message=f"TogetherAIException - {error_response['error']}",
            model=model,
            llm_provider="together_ai",
            response=getattr(original_exception, "response", None),
        )
    elif "A timeout occurred" in error_str:
        raise Timeout(
            message=f"TogetherAIException - {error_str}",
            model=model,
            llm_provider="together_ai",
        )
    elif (
        "error" in error_response
        and "API key doesn't match expected format." in error_response["error"]
        or "error_type" in error_response
        and error_response["error_type"] == "validation"
    ):
        raise BadRequestError(
            message=f"TogetherAIException - {error_response['error']}",
            model=model,
            llm_provider="together_ai",
            response=getattr(original_exception, "response", None),
        )
    if hasattr(original_exception, "status_code"):
        if original_exception.status_code == 408:

View on GitHub (pinned to 6c2dcb801b)

Solutions

1. Retry with backoff — timeouts are frequently transient
2. Reduce max_tokens or prompt size to fit the time budget
3. Set num_retries=3 on the call
4. Try a smaller/faster Together model or off-peak scheduling

Example fix

# before
litellm.completion(model="together_ai/llama-2-70b", messages=msgs, max_tokens=2048)

# after
litellm.completion(
    model="together_ai/llama-2-70b", messages=msgs,
    max_tokens=512, num_retries=3,
)

def is_together_timeout(e: Exception) -> bool:
    return (
        isinstance(e, litellm.Timeout)
        and getattr(e, "llm_provider", "") == "together_ai"
    )

try:
    resp = litellm.completion(model=m, messages=msgs, num_retries=3, retry_after=2)
except litellm.Timeout:
    logger.warning("Together AI timed out")
    raise

When it happens

Trigger: Long generations on big Together models (70B-class) that exceed Together's server-side time budget; overloaded queues for popular models; huge prompts slowing prefill.

Common situations: High max_tokens on 70B models at peak hours; retry loops amplifying load; batch inference without time budgets.

Related errors

• TogetherAIException - {original_exception.message}
• Task {task_id} did not complete within {max_attempts * poll_
• BedrockException: Timeout Error - {error_str}
• TogetherAIException - {error_response['error']}
• APITimeoutError - Request timed out. Error_str: {error_str}

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/82881b1f3a2c843a. Report an issue: GitHub.