unslothai/unsloth · error · RuntimeError

text_encoder_quant='{requested}' could not be used: {reason}

Error message

text_encoder_quant='{requested}' could not be used: {reason}. Leave it unset to keep the dense bf16 encoder.

What it means

Raised when text_encoder_quant requests a torchao-quantized text encoder while the load's memory mode places that encoder under CPU offload. Offload hooks move modules with Module.to(), and torchao-quantized tensors do not survive that move, so the combination is refused up front (auto_available=False - text_encoder_quant has no Auto mode, so the remedy is to leave it unset). Layerwise fp8 is a dtype cast and unaffected.

Source

Thrown at studio/backend/core/inference/diffusion.py:1264

        elif te_effective is not None and not te_quant_supported(target, te_effective):
            te_reason = (
                "this device does not have the tensor cores that backend needs (a CUDA GPU in "
                "bf16, plus fp8 / int8 / NVFP4 support depending on the mode)"
            )
        elif te_quant_needs_resident_weights(te_effective) and _memory_request_forces_offload(
            memory_mode, cpu_offload
        ):
            # Same fence as the dense transformer above, on the encoder: offload hooks move
            # modules with Module.to(), torchao tensors do not survive it, and the loader reports
            # those modes unsupported once offload is active -- after the resident pipeline is
            # already gone. Layerwise fp8 is a dtype cast and is unaffected.
            requested_memory = normalize_memory_mode(memory_mode) or "cpu_offload"
            te_reason = (
                f"'{requested_memory}' memory places the text encoder under CPU offload, and "
                "torchao quantised tensors cannot be moved by the offload hooks"
            )
        if te_reason is not None:
            raise RuntimeError(
                precision_refusal_message(
                    "text_encoder_quant",
                    te_mode,
                    te_reason,
                    off_label = "leave it unset to keep the dense bf16 encoder",
                    auto_available = False,
                )
            )

    def _resolve_device_target(
        self,
        fam: Optional[DiffusionFamily],
        *,
        ordinal: Optional[int] = None,
    ) -> DiffusionDeviceTarget:
        """The device target with the family fp16 guard applied.

        Routes through _pick_device_and_dtype() (so a monkeypatched override still

View on GitHub (pinned to 203007d190)

Solutions

Remove text_encoder_quant from the request - the encoder then stays dense bf16, which offload can move safely.
Switch the quantization to layerwise fp8 (a dtype cast, not torchao), which is unaffected by the offload hooks.
If the torchao encoder quant is essential, choose a memory mode that keeps the encoder resident (no CPU offload), which may require a smaller model or more VRAM.
Keep transformer_quant instead - the fence applies only to the text encoder side.

Example fix

# before: torchao encoder quant + offload -> RuntimeError
manager.load(repo_id="unsloth/FLUX.1-dev",
            memory_mode="cpu_offload", text_encoder_quant="int8")

# after: leave the encoder dense
manager.load(repo_id="unsloth/FLUX.1-dev",
            memory_mode="cpu_offload", text_encoder_quant=None)

Defensive patterns

Strategy: validation

Validate before calling

def encoder_quant_ok(text_encoder_quant, memory_mode) -> bool:
    """torchao encoder tensors cannot survive CPU-offload Module.to() moves."""
    effective = memory_mode or "cpu_offload"  # unset defaults to cpu_offload
    return text_encoder_quant in (None, "", "off") or "fp8_layerwise" in text_encoder_quant or effective not in ("cpu_offload", "sequential", "low_vram")

Try / catch

try:
    manager.load(repo_id=repo, memory_mode=m, text_encoder_quant=q)
except RuntimeError as e:
    if str(e).startswith("text_encoder_quant="):
        manager.load(repo_id=repo, memory_mode=m, text_encoder_quant=None)  # dense bf16 encoder
    else:
        raise

Prevention

Never combine a torchao text_encoder_quant with cpu_offload/low_vram modes - disable one in the UI when the other is set.
There is no Auto for text_encoder_quant: leave it unset rather than guessing.
Prefer layerwise fp8 when you must quantize the encoder on an offloading host.
Remember this fence exists because the failure would otherwise land after the resident pipeline is already gone - do not bypass it.

When it happens

Trigger: Calling a load with text_encoder_quant set to a torchao scheme (e.g. 'int8', 'fp8_dynamic') while memory_mode is an offload mode (or defaults to cpu_offload when unset) and the encoder must offload - the fence checks the effective memory mode with cpu_offload in it.

Common situations: Low-VRAM GPU (8-12GB) where users enable both aggressive quantization and CPU offload to fit a model; leaving memory_mode unset so it defaults to cpu_offload while also setting text_encoder_quant; copying a config from a high-VRAM machine to a small one.

Related errors

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/f26ba113b25032b2. Report an issue: GitHub.