hiyouga/LlamaFactory · error · ValueError

Unsupported Flash Linear Attention kernels: {sorted(unsuppor

Error message

Unsupported Flash Linear Attention kernels: {sorted(unsupported)}

What it means

The FLA plugin restricts kernel selection to the known set FLASH_LINEAR_ATTENTION_KERNELS. If any requested name is not in that registry the plugin raises a ValueError listing the unsupported names, rather than partially applying an unknown op.

Source

Thrown at src/llamafactory/v1/plugins/model_plugins/kernels/ops/linear_attention/fla.py:79

    def _apply(**kwargs) -> HFModel:
        model = kwargs["model"]
        config = kwargs.get("config") or {}
        include_kernels = config.get("include_kernels", "auto")
        chunk_size = config.get("chunk_size", 64)

        if include_kernels == "auto" or include_kernels is True:
            selected = list(FLASH_LINEAR_ATTENTION_KERNELS)
        elif isinstance(include_kernels, str):
            selected = [name.strip() for name in include_kernels.split(",") if name.strip()]
        else:
            raise TypeError("kernel_config.include_kernels must be 'auto' or a comma-separated string.")

        if not selected:
            raise ValueError("kernel_config.include_kernels must select at least one FLA kernel.")

        unsupported = set(selected).difference(FLASH_LINEAR_ATTENTION_KERNELS)
        if unsupported:
            raise ValueError(f"Unsupported Flash Linear Attention kernels: {sorted(unsupported)}")
        if isinstance(chunk_size, bool) or not isinstance(chunk_size, int) or chunk_size not in SUPPORTED_CHUNK_SIZES:
            raise ValueError(f"chunk_size must be one of {SUPPORTED_CHUNK_SIZES}, got {chunk_size!r}.")

        from fsdp_turbo.ops.registry import get_op
        from fsdp_turbo.utils.patch import patch_model_members

        patched = 0
        named_modules = tuple(model.named_modules())
        for op_name in selected:
            module_attribute = FLA_MODULE_ATTRIBUTES[op_name]
            op = get_op(op_name)
            configured_op = partial(op, chunk_size=chunk_size) if op_name == CHUNK_GATED_DELTA_RULE else op
            targets = {
                f"{type(module).__module__}.{type(module).__name__}.{module_attribute}"
                for _, module in named_modules
                if callable(getattr(module, module_attribute, None))
            }
            matched = patch_model_members(model, sorted(targets), configured_op) if targets else 0

View on GitHub (pinned to f28afaf635)

Solutions

  1. Read the error message — it lists the exact unsupported names; fix typos
  2. Check FLASH_LINEAR_ATTENTION_KERNELS in your installed fla.py to see the accepted names for your version
  3. Set include_kernels to "auto" to select all supported kernels without naming them
  4. Pin or upgrade fsdp_turbo/fla so kernel names match your config

Example fix

# before
kernel_config:
  include_kernels: "chunked_gated_delta_rule"

# after
kernel_config:
  include_kernels: "chunk_gated_delta_rule"
Defensive patterns

Strategy: validation

Validate before calling

from llamafactory.v1.plugins.model_plugins.kernels.ops.linear_attention.fla import FLASH_LINEAR_ATTENTION_KERNELS
bad = set(selected).difference(FLASH_LINEAR_ATTENTION_KERNELS)
if bad:
    raise ValueError(f"unknown FLA kernels: {sorted(bad)}; valid: {sorted(FLASH_LINEAR_ATTENTION_KERNELS)}")

Prevention

When it happens

Trigger: Passing include_kernels containing kernel names that are not in FLASH_LINEAR_ATTENTION_KERNELS — typos, renamed kernels from a different fsdp_turbo/FLA version, or kernels from an unrelated plugin.

Common situations: Version skew: kernel names changed between fsdp_turbo/fla releases but the config kept old names; copy-paste from docs for a different version; typos like 'chunked_delta_rule' vs 'chunk_gated_delta_rule'.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/0febf074905a9f87. Report an issue: GitHub.