hiyouga/LlamaFactory · error · ValueError

cross_entropy and fused_linear_cross_entropy cannot both be

Error message

cross_entropy and fused_linear_cross_entropy cannot both be enabled.

What it means

Within Liger's togglable ops, cross_entropy and fused_linear_cross_entropy are mutually exclusive strategies for the LM head + loss (fused skips materializing logits). Enabling both in one use_kernels list is contradictory and raises ValueError before patching.

Source

Thrown at src/llamafactory/v1/plugins/model_plugins/kernels/liger_kernel_ops.py:122

                "rmsnorm": "rms_norm",
                "flce": "fused_linear_cross_entropy",
                "lce": "fused_linear_cross_entropy",
                "fused_ce": "fused_linear_cross_entropy",
            }
            return aliases.get(key, key)

        if use_kernels is not None and len(use_kernels) == 0:
            return model

        if use_kernels != "auto":
            selected = {_normalize_op_name(k) for k in use_kernels}
            ops = selected - set(togglable)
            if ops:
                raise ValueError(
                    f"Unknown Liger op(s) {sorted(ops)} for model_type={model_type}. Valid: {sorted(togglable)}"
                )
            if "cross_entropy" in selected and "fused_linear_cross_entropy" in selected:
                raise ValueError("cross_entropy and fused_linear_cross_entropy cannot both be enabled.")
            call_kwargs = {name: (name in selected) for name in togglable}
            call_kwargs["model"] = model
        else:
            # Mirror ``liger_kernel`` signature defaults so patches match upstream defaults
            # and logging reflects enabled ops (omitted kwargs only live in the callee).
            call_kwargs = {"model": model}
            for name in togglable:
                param = sig[name]
                if param.default is not inspect.Parameter.empty:
                    call_kwargs[name] = param.default

        if require_logits and "fused_linear_cross_entropy" in sig:
            logger.warning_rank0("Current training stage does not support chunked cross entropy.")
            call_kwargs["fused_linear_cross_entropy"] = False
            call_kwargs["cross_entropy"] = True

        apply_liger_kernel(**call_kwargs)

View on GitHub (pinned to f28afaf635)

Solutions

Keep exactly one: fused_linear_cross_entropy for fused speed (when logits not needed), cross_entropy otherwise.
If require_logits is true, prefer plain cross_entropy — the code already forces non-fused CE in that case.
Prefer use_kernels="auto" which resolves defaults without conflicts.

Example fix

# before
use_kernels = ["cross_entropy", "fused_ce"]

# after
use_kernels = ["fused_linear_cross_entropy"]

Defensive patterns

Strategy: validation

Validate before calling

ALIASES = {'lce': 'fused_linear_cross_entropy', 'fused_ce': 'fused_linear_cross_entropy'}
norm = {ALIASES.get(k, k) for k in use_kernels}
assert not ({'cross_entropy', 'fused_linear_cross_entropy'} <= norm), 'mutually exclusive CE ops both enabled'

Type guard

def ce_ops_consistent(ops: list[str]) -> bool:
    """True when at most one of the two cross-entropy strategies is selected."""
    norm = {ALIASES.get(k, k) for k in ops}
    return not ({'cross_entropy', 'fused_linear_cross_entropy'} <= norm)

Try / catch

try:
    model = KernelPlugin('liger_kernel').apply(model=model, use_kernels=ops)
except ValueError as e:
    if 'cannot both be enabled' in str(e):
        ops = [o for o in ops if o != 'cross_entropy']
        model = KernelPlugin('liger_kernel').apply(model=model, use_kernels=ops)
    else:
        raise

Prevention

Never enumerate 'all ops' blindly; pick one CE strategy deliberately.
Expand aliases (lce, fused_ce) before validating configs.
When logits are needed (require_logits), choose plain cross_entropy.

When it happens

Trigger: use_kernels containing both names (directly or via aliases, e.g. ["cross_entropy", "fused_ce"]) for a model whose signature exposes both toggles.

Common situations: Assembling an 'all ops on' list by enumerating every signature parameter; merging configs from two sources each enabling one variant; alias confusion (lce/fused_ce resolving to the fused variant).

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/f2c1e64e3855a7ac. Report an issue: GitHub.