hiyouga/LlamaFactory · error · NotImplementedError

Device not supported: {device_name}.

Error message

Device not supported: {device_name}.

What it means

Identical guard to the DPO trainer: after Trainer.__init__ the KTO trainer requires self.accelerator (trainer.py:92), which only exists in modern transformers. On an outdated transformers the attribute is missing and the trainer stops with AttributeError('Please update `transformers`.') instead of failing later during distributed setup.

Source

Thrown at scripts/stat_utils/cal_mfu.py:98

    if include_flashattn:
        total_flops += sdpa_flops

    return total_flops


def compute_device_flops(world_size: int) -> float:
    r"""Calculate the FLOPs of the device capability per second."""
    device_name = torch.cuda.get_device_name()
    if "H100" in device_name or "H800" in device_name:
        return 989 * 1e12 * world_size
    elif "A100" in device_name or "A800" in device_name:
        return 312 * 1e12 * world_size
    elif "V100" in device_name:
        return 125 * 1e12 * world_size
    elif "4090" in device_name:
        return 98 * 1e12 * world_size
    else:
        raise NotImplementedError(f"Device not supported: {device_name}.")


def calculate_mfu(
    model_name_or_path: str,
    batch_size: int = 1,
    seq_length: int = 1024,
    num_steps: int = 100,
    finetuning_type: str = "lora",
    flash_attn: str = "auto",
    deepspeed_stage: int = 0,
    disable_gc: bool = False,
    liger_kernel: bool = False,
    unsloth_gc: bool = False,
) -> float:
    r"""Calculate MFU for given model and hyper-params.

    Usage: python cal_mfu.py --model_name_or_path path_to_model --batch_size 1 --seq_length 1024
    """

View on GitHub (pinned to f28afaf635)

Solutions

1. pip install -U transformers (align with LlamaFactory's declared requirements)
2. Confirm the active interpreter/venv actually uses the upgraded version (`python -m pip list | grep transformers`) — multiple envs are a frequent cause

Example fix

# before
transformers 4.3x -> AttributeError: Please update `transformers`.

# after
pip install -U transformers
python -m pip list | grep transformers

import transformers
from packaging.version import parse
assert parse(transformers.__version__) >= parse('4.37'), 'KTO trainer needs modern transformers; pip install -U transformers'

def transformers_new_enough_for_kto() -> bool:
    import transformers
    from packaging.version import parse
    return parse(transformers.__version__) >= parse('4.37.0')

try:
    from llamafactory.train.kto.workflow import run_kto
    run_kto(train_args)
except AttributeError as e:
    if 'update `transformers`' in str(e):
        raise SystemExit('pip install -U transformers') from e
    raise

When it happens

Trigger: Running KTO training (stage: kto) with transformers too old for the Trainer API the code expects; hasattr(self, 'accelerator') is False right after super().__init__.

Common situations: Environment pinned to an old transformers for a different model; partial upgrades where transformers was downgraded by another dependency.

Related errors

• All 'dcp_path', 'hf_path', and 'config_path' are required.
• Both 'hf_path' and 'dcp_path' are required.
• `num_layers` {num_layers} should be divisible by `num_expand
• Qwen2VL requires 3D position ids for mrope.
• Stage does not supported: {stage}.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/4ca0b62d9ccfb263. Report an issue: GitHub.