hiyouga/LlamaFactory · error · NotImplementedError

Unknown attention type: {model_args.flash_attn}

Error message

Unknown attention type: {model_args.flash_attn}

What it means

While resolving the flash-attention implementation in prepare_model_for_training/attn setup, LlamaFactory maps model_args.flash_attn (an AttentionFunction enum) to a transformers attention implementation. Any value that does not match a known enum branch falls into the final else and raises NotImplementedError. In practice this means the config value is not a valid member of the AttentionFunction enum (auto, sdpa, fa2, fa3).

Source

Thrown at src/llamafactory/model/model_utils/attention.py:94

        requested_attn_implementation = "sdpa"
    elif model_args.flash_attn == AttentionFunction.FA2:
        from transformers import is_torch_npu_available

        if not (is_flash_attn_2_available() or is_torch_npu_available()):
            logger.warning_rank0("FlashAttention-2 is not installed.")
            return

        requested_attn_implementation = "flash_attention_2"
    elif model_args.flash_attn == AttentionFunction.FA3:
        from transformers.utils import is_flash_attn_3_available

        if not is_flash_attn_3_available():
            logger.warning_rank0("FlashAttention-3 is not installed.")
            return

        requested_attn_implementation = "flash_attention_3"
    else:
        raise NotImplementedError(f"Unknown attention type: {model_args.flash_attn}")

    if getattr(config, "model_type", None) == "internlm2":  # special case for custom models
        setattr(config, "attn_implementation", requested_attn_implementation)
    elif getattr(config, "model_type", None) == "kimi_vl":
        setattr(config.vision_config, "_attn_implementation", requested_attn_implementation)
        setattr(config.text_config, "_attn_implementation", requested_attn_implementation)
    elif getattr(config, "model_type", None) == "youtu_vl":
        setattr(config, "attn_implementation", requested_attn_implementation)
        setattr(config, "_attn_implementation", requested_attn_implementation)
        if hasattr(config, "vision_config"):
            setattr(config.vision_config, "_attn_implementation", requested_attn_implementation)
        if hasattr(config, "text_config"):
            setattr(config.text_config, "_attn_implementation", requested_attn_implementation)
    else:
        setattr(config, "_attn_implementation", requested_attn_implementation)


def print_attn_implementation(config: "PretrainedConfig") -> None:

View on GitHub (pinned to f28afaf635)

Solutions

1. Set flash_attn to a valid AttentionFunction value: auto, sdpa, fa2 (or fa3 if your build has it).
2. If you passed a string from Python, import and use the enum: from llamafactory.extras.constants import AttentionFunction; flash_attn=AttentionFunction.FA2.
3. Check the enum definition in src/llamafactory/extras/constants.py for the exact accepted names in your installed version.
4. Update or align your YAML config with the installed LlamaFactory version.

Example fix

# before (yaml)
flash_attn: flash_attn_2   # -> NotImplementedError: Unknown attention type

# after (yaml)
flash_attn: fa2

from llamafactory.extras.constants import AttentionFunction

valid = {e.value for e in AttentionFunction}
assert flash_attn in valid, f"flash_attn must be one of {valid}, got {flash_attn!r}"

from llamafactory.extras.constants import AttentionFunction

def is_valid_attention(value: str) -> bool:
    return value in {e.value for e in AttentionFunction}

When it happens

Trigger: Setting flash_attn: <something not in the AttentionFunction enum> in the YAML model_args (e.g. a typo like 'flash_attn_2', 'FA2', 'xformers'), or passing an outdated/new enum value across a version mismatch between config files and the installed LlamaFactory. The function dispatches on model_args.flash_attn == AttentionFunction.FA2 / FA3 and raises in the else branch.

Common situations: Copy-pasted YAML from an older/newer LlamaFactory version using a different enum spelling; passing a raw string via Python API instead of the enum; a user requesting 'flash_attention_3' on a build whose enum lacks FA3.

Related errors

• PPO training is incompatible with S^2-Attn.
• Please upgrade `transformers` to 4.34.0
• Unable to process key {key}
• bf16 and fp16 cannot be both True.
• Unsupported model type: {getattr(config, 'model_type')}.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/14f39b23a7cfbdbb. Report an issue: GitHub.