hiyouga/LlamaFactory · error · ValueError

Unsupported `flash_attn`: {self.flash_attn}. Supported value

Error message

Unsupported `flash_attn`: {self.flash_attn}. Supported values are: {supported_flash_attn}.

What it means

`ModelArguments.__post_init__` in the v1 config validates `flash_attn` against the values of the `AttentionFunction` enum and raises this ValueError for anything else. The supported set comes from `AttentionFunction` members (e.g. `auto`, `sdpa`, `flash_attention_2`, `full`).

Source

Thrown at src/llamafactory/v1/config/model_args.py:66

        metadata={"help": "Initialization configuration for the model."},
    )
    peft_config: PluginConfig | None = field(
        default=None,
        metadata={"help": "PEFT configuration for the model."},
    )
    kernel_config: PluginConfig | None = field(
        default=None,
        metadata={"help": "Kernel configuration for the model."},
    )
    quant_config: PluginConfig | None = field(
        default=None,
        metadata={"help": "Quantization configuration for the model."},
    )

    def __post_init__(self) -> None:
        supported_flash_attn = [item.value for item in AttentionFunction]
        if self.flash_attn not in supported_flash_attn:
            raise ValueError(
                f"Unsupported `flash_attn`: {self.flash_attn}. Supported values are: {supported_flash_attn}."
            )

        self.init_config = get_plugin_config(self.init_config)
        self.peft_config = get_plugin_config(self.peft_config)
        self.kernel_config = get_plugin_config(self.kernel_config)
        self.quant_config = get_plugin_config(self.quant_config)

View on GitHub (pinned to f28afaf635)

Solutions

Use one of the values printed in the error's `supported_flash_attn` list, typically `flash_attention_2`
Install/verify flash-attn is available if you pick `flash_attention_2`
If unsure, use `auto` or `sdpa` which need no extra kernels

Example fix

# before (yaml)
model:
  flash_attn: fa2

# after (yaml)
model:
  flash_attn: flash_attention_2

Defensive patterns

Strategy: validation

Validate before calling

from llamafactory.v1.config import AttentionFunction  # enum module path

SUPPORTED = {item.value for item in AttentionFunction}

def validate_flash_attn(value: str) -> str:
    if value not in SUPPORTED:
        raise SystemExit(f"flash_attn must be one of {sorted(SUPPORTED)}, got {value}")
    return value

Type guard

def is_supported_flash_attn(value: object) -> bool:
    return isinstance(value, str) and value in {i.value for i in AttentionFunction}

Prevention

Use exact enum strings (`flash_attention_2`, `sdpa`, `auto`) from AttentionFunction
Reject booleans in config validation — flash_attn is a string enum in v1
Add a JSON-schema/enum constraint to shared config tooling

When it happens

Trigger: Setting `flash_attn: fa2`, `flash_attn: flash_attention`, `flash_attn: true`, or any string not in the enum when constructing v1 `ModelArguments` or training via the v1 launcher.

Common situations: Copying v0-era shorthand (`fa2`) or boolean values from old configs; version drift where a renamed enum value no longer exists; typos.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/c707ce8865e70653. Report an issue: GitHub.