unslothai/unsloth · error · RuntimeError

text_encoder_quant='{requested}' could not be used: {reason}

Error message

text_encoder_quant='{requested}' could not be used: {reason}. Leave it unset to keep the dense bf16 encoder.

What it means

The text_encoder_quant gate refuses a pinned encoder quantization in two situations: the device lacks the tensor cores the mode needs (no CUDA GPU with the required bf16/fp8/int8/NVFP4 support), or the memory mode forces CPU offload while the mode requires resident weights (torchao tensors cannot be moved by the offload hooks; layerwise fp8 is exempt as a plain dtype cast). Auto is not offered here — the fallback is simply leaving it unset.

Source

Thrown at studio/backend/core/inference/video.py:344

        # Same as the image gate: the casters import torchao only after the pipeline is built.
        te_reason = (
            "torchao is not importable on this install, and these encoder modes are torchao "
            "quantisations"
        )
    elif te_effective is not None and not te_quant_supported(target, te_effective):
        te_reason = (
            "this device does not have the tensor cores that backend needs (a CUDA GPU in "
            "bf16, plus fp8 / int8 / NVFP4 support depending on the mode)"
        )
    elif te_quant_needs_resident_weights(te_effective) and forces_offload:
        # Same fence on the encoder: the loader reports those modes unsupported once offload is
        # active, and by then the resident model is gone. Layerwise fp8 is a dtype cast.
        te_reason = (
            f"'{normalize_memory_mode(memory_mode)}' memory places the text encoder under CPU "
            "offload, and torchao quantised tensors cannot be moved by the offload hooks"
        )
    if te_reason is not None:
        raise RuntimeError(
            precision_refusal_message(
                "text_encoder_quant",
                te_mode,
                te_reason,
                off_label = "leave it unset to keep the dense bf16 encoder",
                auto_available = False,
            )
        )


def _is_trusted_video_repo(repo_id: str) -> bool:
    """Whether a NON-GGUF load may deserialise this repo (see the image twin)."""
    try:
        if Path(repo_id).expanduser().exists():
            return True
    except OSError:
        pass
    rid = repo_id.strip().lower()

View on GitHub (pinned to 203007d190)

Solutions

1. Leave text_encoder_quant unset to keep the dense bf16 encoder — the always-safe option.
2. If you need encoder quant, switch memory_mode so the text encoder stays resident (no CPU offload).
3. On non-CUDA hosts, drop encoder quant entirely; only CUDA GPUs with the right tensor cores support these modes.

Example fix

# before
load_video_model(repo_id=..., text_encoder_quant='int8', memory_mode='low')

# after
load_video_model(repo_id=..., text_encoder_quant=None, memory_mode='low')

# before requesting: encoder quant needs CUDA tensor cores AND resident weights
import torch
def encoder_quant_safe(requested, memory_mode_forces_offload):
    if requested in (None, ''):
        return True
    if not (torch.cuda.is_available()):
        return False
    return not memory_mode_forces_offload  # layerwise fp8 excepted server-side

try:
    load_video_model(repo_id=r, text_encoder_quant=te, memory_mode=mm)
except RuntimeError as e:
    if 'text_encoder_quant' in str(e):
        load_video_model(repo_id=r, text_encoder_quant=None, memory_mode=mm)
    else:
        raise

When it happens

Trigger: Passing text_encoder_quant with a non-NVIDIA or older GPU (te_reason set by the tensor-core branch), or combining an offload-forcing memory_mode with a quant mode flagged by te_quant_needs_resident_weights (te_effective).

Common situations: Enabling encoder quant on Apple Silicon or AMD; setting an aggressive memory_mode ('low' style offload) together with int8/NVFP4 encoder quant; configs copied from a 24 GB+ GPU host to a small-VRAM machine where offload is auto-selected.

Related errors

• transformer_quant='{requested}' could not be used: {reason}.
• '{fam.name}' cannot run on Apple Silicon: its Modular Diffus
• transformer_quant '{requested_scheme}' is unavailable for '{
• This LTX checkpoint stores scaled fp8 weights, which this lo
• transformer_quant='{requested}' could not be used: {reason}.

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/51838a81d0fb6d7a. Report an issue: GitHub.