unslothai/unsloth · error · ValueError

This trainer requires a bfloat16-capable GPU (Ampere or newe

Error message

This trainer requires a bfloat16-capable GPU (Ampere or newer); this CUDA device does not support bf16.

What it means

Raised when the run resolves to a CUDA device that lacks native bf16 support, i.e. compute capability major < 8 (pre-Ampere). The whole flow-matching + 4-bit path is bf16 on GPU, and the check deliberately uses native_bf16_supported() rather than is_bf16_supported() because the latter counts emulation. CPU-only hosts are exempt — they run fp32 so import/unit tests stay architecture-agnostic.

Source

Thrown at studio/backend/core/training/diffusion_dit_trainer.py:1870

    save_on_stop = True

    def _check_stop() -> bool:
        nonlocal save_on_stop
        if should_stop is None:
            return False
        sig = should_stop()
        if not sig:
            return False
        if isinstance(sig, dict) and sig.get("save") is False:
            save_on_stop = False
        return True

    device = "cuda" if torch.cuda.is_available() else "cpu"
    # The flow-matching + 4-bit path is bf16 throughout (fp32 on a CPU-only box, to keep import/unit tests architecture-agnostic).
    # Fail fast on pre-Ampere CUDA, gating on NATIVE bf16 (capability major >= 8), since is_bf16_supported() counts emulation.
    if device == "cuda" and not native_bf16_supported():
        raise ValueError(
            "This trainer requires a bfloat16-capable GPU (Ampere or newer); "
            "this CUDA device does not support bf16."
        )
    weight_dtype = torch.bfloat16 if device == "cuda" else torch.float32

    _assert_trusted_base_model(cfg.base_model)
    # The repo this run will FETCH, which is what the start route preflights. Checking the
    # canonical id instead would raise here for a gated base that normalization already
    # redirected to its ungated mirror -- after the route had answered 200 and freed the
    # resident models, so the request fails as a dead job rather than as a fast 400.
    _assert_gated_access(cfg.fetch_base_model or cfg.base_model, cfg.hf_token)
    pairs = discover_image_caption_pairs(
        cfg.data_dir, instance_prompt = cfg.instance_prompt, caption_column = cfg.caption_column
    )
    # Resolve num_epochs into a concrete train_steps now the dataset size is known, and rebind cfg so every downstream read agrees.
    cfg = replace(cfg, train_steps = resolve_train_steps(cfg, len(pairs)), num_epochs = 0)
    # Validate a resume request against this run's identity BEFORE the multi-GB phased load, so
    # a mismatched checkpoint fails in seconds. The identity uses the RESOLVED LoRA targets,

View on GitHub (pinned to 203007d190)

Solutions

1. Move the run to an Ampere-or-newer GPU (RTX 30xx/40xx/50xx, A-series, H-series).
2. If stuck on this host, run CPU-only (device resolves to 'cpu' and trains fp32) — practical only for smoke tests, not real training.
3. Confirm the visible device: check nvidia-smi and CUDA_VISIBLE_DEVICES so an older secondary GPU is not being picked.

Example fix

# before
# training on a GTX 1080 Ti (sm61) -> ValueError on startup

# after
CUDA_VISIBLE_DEVICES=1 python train.py  # point at the RTX 3090 (sm86) instead

import torch

def native_bf16_gpu() -> bool:
    if not torch.cuda.is_available():
        return False
    try:
        return torch.cuda.get_device_capability()[0] >= 8
    except Exception:
        return False

When it happens

Trigger: Training on GTX 10xx/16xx, Titan V, or other Turing/Pascal/Volta cards where device=='cuda' but capability < (8, 0); is_bf16_supported() lying via emulation on such cards.

Common situations: Older gaming rigs (GTX 1080, RTX 20-series); T4 instances on cheap cloud tiers (Turing, no native bf16); VMs passthrough-assigning an old GPU; free Colab/Kaggle legacy GPU runtimes.

Related errors

• STT model loading was cancelled so training could start.
• base_precision={mode!r} needs a CUDA GPU; this host has none
• '{resolved_family}' LoRA training requires bf16: fp16 overfl
• Invalid gpu_ids {requested_ids}: duplicate GPU IDs are not a
• Invalid gpu_ids {requested_ids}: GPU IDs must be non-negativ

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/05d29304a098538f. Report an issue: GitHub.