{"record":{"id":"14f39b23a7cfbdbb","repo":"hiyouga/LlamaFactory","slug":"unknown-attention-type-model-args-flash-attn","errorCode":null,"errorMessage":"Unknown attention type: {model_args.flash_attn}","messagePattern":"Unknown attention type: (.+?)","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/model/model_utils/attention.py","lineNumber":94,"sourceCode":"        requested_attn_implementation = \"sdpa\"\n    elif model_args.flash_attn == AttentionFunction.FA2:\n        from transformers import is_torch_npu_available\n\n        if not (is_flash_attn_2_available() or is_torch_npu_available()):\n            logger.warning_rank0(\"FlashAttention-2 is not installed.\")\n            return\n\n        requested_attn_implementation = \"flash_attention_2\"\n    elif model_args.flash_attn == AttentionFunction.FA3:\n        from transformers.utils import is_flash_attn_3_available\n\n        if not is_flash_attn_3_available():\n            logger.warning_rank0(\"FlashAttention-3 is not installed.\")\n            return\n\n        requested_attn_implementation = \"flash_attention_3\"\n    else:\n        raise NotImplementedError(f\"Unknown attention type: {model_args.flash_attn}\")\n\n    if getattr(config, \"model_type\", None) == \"internlm2\":  # special case for custom models\n        setattr(config, \"attn_implementation\", requested_attn_implementation)\n    elif getattr(config, \"model_type\", None) == \"kimi_vl\":\n        setattr(config.vision_config, \"_attn_implementation\", requested_attn_implementation)\n        setattr(config.text_config, \"_attn_implementation\", requested_attn_implementation)\n    elif getattr(config, \"model_type\", None) == \"youtu_vl\":\n        setattr(config, \"attn_implementation\", requested_attn_implementation)\n        setattr(config, \"_attn_implementation\", requested_attn_implementation)\n        if hasattr(config, \"vision_config\"):\n            setattr(config.vision_config, \"_attn_implementation\", requested_attn_implementation)\n        if hasattr(config, \"text_config\"):\n            setattr(config.text_config, \"_attn_implementation\", requested_attn_implementation)\n    else:\n        setattr(config, \"_attn_implementation\", requested_attn_implementation)\n\n\ndef print_attn_implementation(config: \"PretrainedConfig\") -> None:","sourceCodeStart":76,"sourceCodeEnd":112,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/model/model_utils/attention.py#L76-L112","documentation":"While resolving the flash-attention implementation in prepare_model_for_training/attn setup, LlamaFactory maps model_args.flash_attn (an AttentionFunction enum) to a transformers attention implementation. Any value that does not match a known enum branch falls into the final else and raises NotImplementedError. In practice this means the config value is not a valid member of the AttentionFunction enum (auto, sdpa, fa2, fa3).","triggerScenarios":"Setting flash_attn: <something not in the AttentionFunction enum> in the YAML model_args (e.g. a typo like 'flash_attn_2', 'FA2', 'xformers'), or passing an outdated/new enum value across a version mismatch between config files and the installed LlamaFactory. The function dispatches on model_args.flash_attn == AttentionFunction.FA2 / FA3 and raises in the else branch.","commonSituations":"Copy-pasted YAML from an older/newer LlamaFactory version using a different enum spelling; passing a raw string via Python API instead of the enum; a user requesting 'flash_attention_3' on a build whose enum lacks FA3.","solutions":["Set flash_attn to a valid AttentionFunction value: auto, sdpa, fa2 (or fa3 if your build has it).","If you passed a string from Python, import and use the enum: from llamafactory.extras.constants import AttentionFunction; flash_attn=AttentionFunction.FA2.","Check the enum definition in src/llamafactory/extras/constants.py for the exact accepted names in your installed version.","Update or align your YAML config with the installed LlamaFactory version."],"exampleFix":"# before (yaml)\nflash_attn: flash_attn_2   # -> NotImplementedError: Unknown attention type\n\n# after (yaml)\nflash_attn: fa2","handlingStrategy":"type-guard","validationCode":"from llamafactory.extras.constants import AttentionFunction\n\nvalid = {e.value for e in AttentionFunction}\nassert flash_attn in valid, f\"flash_attn must be one of {valid}, got {flash_attn!r}\"","typeGuard":"from llamafactory.extras.constants import AttentionFunction\n\ndef is_valid_attention(value: str) -> bool:\n    return value in {e.value for e in AttentionFunction}","tryCatchPattern":null,"preventionTips":["Always source flash_attn values from the AttentionFunction enum instead of hand-typed strings.","Validate YAML model_args against the installed version's constants before launching a run."],"tags":["flash-attention","config","enum","attention"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}