oobabooga/textgen · error · ModuleNotFoundError

Failed to import 'tensorrt_llm'. Please install it manually

Error message

Failed to import 'tensorrt_llm'. Please install it manually following the instructions in the TensorRT-LLM GitHub repository.

What it means

Error "Failed to import 'tensorrt_llm'. Please install it manually following the instructions in the TensorRT-LLM GitHub repository." thrown in oobabooga/textgen.

Source

Thrown at modules/models.py:128

def ExLlamav3_HF_loader(model_name):
    from modules.exllamav3_hf import Exllamav3HF

    return Exllamav3HF.from_pretrained(model_name)


def ExLlamav3_loader(model_name):
    from modules.exllamav3 import Exllamav3Model

    model, tokenizer = Exllamav3Model.from_pretrained(model_name)
    return model, tokenizer


def TensorRT_LLM_loader(model_name):
    try:
        from modules.tensorrt_llm import TensorRTLLMModel
    except ModuleNotFoundError:
        raise ModuleNotFoundError("Failed to import 'tensorrt_llm'. Please install it manually following the instructions in the TensorRT-LLM GitHub repository.")

    model = TensorRTLLMModel.from_pretrained(model_name)
    return model, model.tokenizer


def unload_model(keep_model_name=False):
    if shared.model is None:
        return

    model_class_name = shared.model.__class__.__name__
    is_llamacpp = (model_class_name == 'LlamaServer')

    if model_class_name in ['Exllamav3Model', 'Exllamav3HF', 'TensorRTLLMModel']:
        shared.model.unload()
    elif model_class_name == 'LlamaServer':
        shared.model.stop()

    shared.model = shared.tokenizer = None

View on GitHub (pinned to ed888c71f2)

Solutions

1. Install TensorRT-LLM manually following the instructions in its GitHub repository.
2. Select a different model loader that does not require tensorrt_llm.

When it happens

Trigger: Raised when the TensorRT-LLM loader is selected but the 'tensorrt_llm' Python package cannot be imported. Triggers when TensorRT-LLM is not installed in the environment; install it manually following the TensorRT-LLM GitHub repository instructions.

Common situations: See trigger scenarios.

AI-assisted analysis of oobabooga/textgen@ed888c71f2 (2026-08-15). Data as JSON: /api/errors/5686f1831a68c84e. Report an issue: GitHub.