hiyouga/LlamaFactory · error · NotImplementedError

Other sequence parallel modes are to be implemented.

Error message

Other sequence parallel modes are to be implemented.

What it means

The ulysses sequence-parallel plugin only implements one attention path (the flash-attention branch shown); when the code falls through to the else branch — any sequence parallel mode other than the supported Ulysses/flash-attn combination — it raises NotImplementedError explicitly, signalling planned-but-missing support.

Source

Thrown at src/llamafactory/v1/plugins/model_plugins/parallelization/sequence_parallel.py:79

    target_dtype=None,
    **kwargs,
):
    if mode == "ulysses":
        dist_attn = UlyssesAttention(sequence_process_group=group, attn_fn=attn_fn)
        attn_output = dist_attn(
            query_states,
            key_states,
            value_states,
            attention_mask,
            query_length=query_states.shape[1] * sequence_parallel_size,
            deterministic=deterministic,
            dropout_p=dropout,
            causal=is_causal,
            position_ids=kwargs.get("position_ids", None),
            target_dtype=target_dtype,
        )
    else:
        raise NotImplementedError("Other sequence parallel modes are to be implemented.")

    return attn_output


@SequenceParallelModelPlugin("ulysses").register()
def apply_sequence_parallel(model, cp_size: int):
    # Replace _flash_attention_forward with new_flash_attn_forward
    module = sys.modules[model.__module__]

    set_ulysses_sequence_parallel_group(DistributedInterface().get_group(Dim.CP))

    try:
        num_attention_heads, num_key_value_heads = (
            model.config.num_attention_heads,
            model.config.num_key_value_heads,
        )
    except AttributeError:
        num_attention_heads, num_key_value_heads = (

View on GitHub (pinned to f28afaf635)

Solutions

1. Set the attention implementation to flash_attention_2 when using ulysses sequence parallelism
2. Reduce/disable sequence parallelism (cp_size = 1) if you must use sdpa/eager
3. Track the upstream repo for additional SP modes; do not attempt to bypass the raise — other paths are genuinely unimplemented

Example fix

# before
compute_strategy:
  cp_size: 4
flash_attn: fa3  # or attention left as sdpa

# after
compute_strategy:
  cp_size: 4
  use_flash_attention: true  # ulysses requires the flash-attn path

if cp_size > 1 and not config.get("use_flash_attention", False):
    raise ValueError("ulysses sequence parallel requires flash_attention_2; set it or use cp_size=1")

When it happens

Trigger: Enabling sequence parallelism with an attention implementation other than flash_attention_2 (e.g. sdpa/eager) or a CP mode the plugin does not implement, so control reaches the else branch in the patched attention forward.

Common situations: Setting compute_strategy: use_flash_attention off while cp_size > 1; mixing sequence parallel with sdpa or eager attention; config templates that enable ulysses regardless of attention backend.

Related errors

• DPO trainer currently only supports cp_size == 1.
• RM trainer currently only supports cp_size == 1.
• Invalid role
• SGLang only supports n=1.
• SGLang engine does not support `get_scores`.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/c01b934c54c59fc4. Report an issue: GitHub.