BerriAI/litellm · error · Exception

Importing torch, transformers, petals failed Try pip install

Error message

Importing torch, transformers, petals failed
Try pip installing petals 
pip install git+https://github.com/bigscience-workshop/petals

What it means

When no api_base is configured, LiteLLM's Petals backend runs inference locally by importing petals (AutoDistributedModelForCausalLM) and transformers (AutoTokenizer), which in turn require torch. If any of those imports fail, it raises this generic Exception with pip install instructions pointing at the bigscience-workshop petals GitHub repo. This is a client-side dependency failure, not an API error.

Source

Thrown at litellm/llms/petals/completion/handler.py:95

            additional_args={"complete_input_dict": optional_params},
        )

        ## RESPONSE OBJECT
        try:
            output_text = response.json()["outputs"]
        except Exception as e:
            PetalsError(
                status_code=response.status_code,
                message=str(e),
                headers=response.headers,
            )

    else:
        try:
            from petals import AutoDistributedModelForCausalLM
            from transformers import AutoTokenizer
        except Exception:
            raise Exception(
                "Importing torch, transformers, petals failed\nTry pip installing petals \npip install git+https://github.com/bigscience-workshop/petals"
            )

        model = model

        tokenizer: Final = AutoTokenizer.from_pretrained(model, use_fast=False, add_bos_token=False)
        model_obj: Final = AutoDistributedModelForCausalLM.from_pretrained(model)

        ## LOGGING
        logging_obj.pre_call(
            input=prompt,
            api_key="",
            additional_args={"complete_input_dict": optional_params},
        )

        ## COMPLETION CALL
        inputs: Final = tokenizer(prompt, return_tensors="pt")["input_ids"]

View on GitHub (pinned to 77b7c6c40c)

Solutions

pip install git+https://github.com/bigscience-workshop/petals plus transformers and torch
Verify in the same interpreter: python -c 'import petals, transformers, torch'
If the stack will not install (petals is largely unmaintained), point api_base at a hosted Petals server or switch to an API-backed model
Ensure the host has enough RAM (and optionally GPU) for local swarm inference

Example fix

# before
resp = litellm.completion(model="petals/petals-team/StableBeluga2", messages=[...])  # ImportError

# after
# shell:
#   pip install git+https://github.com/bigscience-workshop/petals transformers torch
resp = litellm.completion(model="petals/petals-team/StableBeluga2", messages=[...])

Defensive patterns

Strategy: validation

Validate before calling

import importlib.util

def petals_stack_available() -> bool:
    return all(importlib.util.find_spec(m) is not None for m in ("petals", "transformers", "torch"))

# fail fast before routing traffic to petals models
if not petals_stack_available():
    raise RuntimeError("Install petals/transformers/torch or route petals traffic via api_base")

Try / catch

Catch Exception around litellm.completion for petals/* models and inspect the message for the 'Importing torch, transformers, petals failed' text; convert it into a clear dependency-missing error rather than letting the generic message leak to users.

Prevention

If you route to petals models, pin petals/transformers/torch in requirements
Health-check optional-backend imports at startup
Prefer api_base-backed Petals servers or API-hosted models unless you specifically need local swarm inference

When it happens

Trigger: Calling litellm.completion with a petals/* model while petals, transformers, or torch is not installed (or fails to import) in the active Python interpreter, and no api_base was provided to route to a remote Petals server.

Common situations: Fresh virtualenv missing the heavy ML stack; Python version incompatible with available torch wheels; petals being unmaintained so plain 'pip install petals' no longer resolves - hence the git+https instruction; developers unaware petals models run locally via swarm inference.

Understand the failure class

Background: "not installed", "pip install", "required for": how missing-dependency errors surface across open-source libraries — this error's family across 34 libraries.

Related errors

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/39cbfc06eb3deede. Report an issue: GitHub.