unslothai/unsloth · error · ValueError

'{resolved_family}' LoRA training requires bf16: fp16 overfl

Error message

'{resolved_family}' LoRA training requires bf16: fp16 overflows its fp32 RoPE / embedder internals. Set mixed precision to bf16.

What it means

Raised when mixed_precision='fp16' is requested for a family in _FORCE_BF16_FAMILIES. These bf16-only DiT architectures run their RoPE positional encoding and text embedders in fp32; fp16 overflows those internals and the training diverges or NaNs. The validator refuses the combination up front, before GPU models are evicted.

Source

Thrown at studio/backend/core/training/diffusion_train_common.py:1041

        # A video family's VAE compresses space by 32, so an off-grid resolution changes the
        # latent geometry silently. Refuse it here, before the GPU models are evicted.
        if (
            resolved_family in TRAINABLE_VIDEO_FAMILIES
            and self.resolution % _VIDEO_RESOLUTION_MULTIPLE != 0
        ):
            raise ValueError(
                f"'{resolved_family}' trains at a resolution that is a multiple of "
                f"{_VIDEO_RESOLUTION_MULTIPLE} (its VAE compresses space by that factor); "
                f"got {self.resolution}."
            )
        if self.mixed_precision not in ("bf16", "fp16", "no"):
            raise ValueError("mixed_precision must be one of bf16 / fp16 / no")
        # torch.manual_seed unpacks int64/uint64, so anything wider raises inside the trainer, after eviction. Catch it here.
        if not -(2**63) <= int(self.seed) <= 2**64 - 1:
            raise ValueError("seed must fit in torch's 64-bit range")
        # Refuse fp16 for a bf16-only DiT family up front, before evicting resident models.
        if self.mixed_precision == "fp16" and resolved_family in _FORCE_BF16_FAMILIES:
            raise ValueError(
                f"'{resolved_family}' LoRA training requires bf16: fp16 overflows its fp32 "
                f"RoPE / embedder internals. Set mixed precision to bf16."
            )
        if str(self.lr_scheduler) not in _LR_SCHEDULERS:
            raise ValueError(
                f"lr_scheduler must be one of {', '.join(sorted(_LR_SCHEDULERS))}; "
                f"got {self.lr_scheduler!r}"
            )
        if not 1 <= int(self.cache_variants) <= 16:
            raise ValueError("cache_variants must be between 1 and 16")
        # Checkpointing knobs. Rejected here, before the route evicts resident GPU models, rather than deep in the loop.
        try:
            save_steps = int(self.save_steps or 0)
            save_total_limit = int(self.save_total_limit or 0)
        except (TypeError, ValueError) as exc:
            raise ValueError(
                f"save_steps / save_total_limit must be whole numbers, got "
                f"{self.save_steps!r} / {self.save_total_limit!r}"

View on GitHub (pinned to 203007d190)

Solutions

1. Set mixed_precision='bf16' for this family (the error message says exactly this).
2. If the GPU cannot do bf16 (pre-Ampere NVIDIA), you cannot train this family in fp16 — use mixed_precision='no' (fp32) or switch hardware/base model.
3. Tag per-family precision defaults in your config templates so fp16 is never applied to bf16-only families.

Example fix

# before
config = TrainConfig(base_model='...', mixed_precision='fp16')

# after
config = TrainConfig(base_model='...', mixed_precision='bf16')

def check_family_precision(family, mixed_precision) -> str:
    # Mirror the trainer's gate: bf16-only DiT families refuse fp16.
    if mixed_precision == "fp16" and family in FORCE_BF16_FAMILIES:
        raise ValueError(f"{family} requires bf16; fp16 overflows its RoPE/embedder internals")
    return mixed_precision

def supports_fp16(family) -> bool:
    return family not in FORCE_BF16_FAMILIES

try:
    session.submit_training(config)
except ValueError as e:
    if "requires bf16" in str(e):
        config.mixed_precision = "bf16"
        session.submit_training(config)
    else:
        raise

When it happens

Trigger: Training a bf16-only DiT family (e.g. modern DiT video/image models) with mixed_precision='fp16'. Often happens when a config that worked on an fp16-era UNet model (SD1.5/SDXL-class) is reused unchanged on a DiT family.

Common situations: Porting a known-good SDXL fp16 recipe to a newer DiT base model; choosing fp16 on pre-Ampere GPUs where bf16 is slow/unavailable; hyperparameter sweeps that try all precision options.

Related errors

• mixed_precision must be one of bf16 / fp16 / no
• {spec.family} LoRA training requires bf16: fp16 overflows it
• This trainer requires a bfloat16-capable GPU (Ampere or newe
• gradient_accumulation_steps must be >= 1
• lora_rank must be >= 1

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/f46c746f2e973149. Report an issue: GitHub.