{"record":{"id":"c01b934c54c59fc4","repo":"hiyouga/LlamaFactory","slug":"other-sequence-parallel-modes-are-to-be-implemente","errorCode":null,"errorMessage":"Other sequence parallel modes are to be implemented.","messagePattern":"Other sequence parallel modes are to be implemented\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/plugins/model_plugins/parallelization/sequence_parallel.py","lineNumber":79,"sourceCode":"    target_dtype=None,\n    **kwargs,\n):\n    if mode == \"ulysses\":\n        dist_attn = UlyssesAttention(sequence_process_group=group, attn_fn=attn_fn)\n        attn_output = dist_attn(\n            query_states,\n            key_states,\n            value_states,\n            attention_mask,\n            query_length=query_states.shape[1] * sequence_parallel_size,\n            deterministic=deterministic,\n            dropout_p=dropout,\n            causal=is_causal,\n            position_ids=kwargs.get(\"position_ids\", None),\n            target_dtype=target_dtype,\n        )\n    else:\n        raise NotImplementedError(\"Other sequence parallel modes are to be implemented.\")\n\n    return attn_output\n\n\n@SequenceParallelModelPlugin(\"ulysses\").register()\ndef apply_sequence_parallel(model, cp_size: int):\n    # Replace _flash_attention_forward with new_flash_attn_forward\n    module = sys.modules[model.__module__]\n\n    set_ulysses_sequence_parallel_group(DistributedInterface().get_group(Dim.CP))\n\n    try:\n        num_attention_heads, num_key_value_heads = (\n            model.config.num_attention_heads,\n            model.config.num_key_value_heads,\n        )\n    except AttributeError:\n        num_attention_heads, num_key_value_heads = (","sourceCodeStart":61,"sourceCodeEnd":97,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/plugins/model_plugins/parallelization/sequence_parallel.py#L61-L97","documentation":"The ulysses sequence-parallel plugin only implements one attention path (the flash-attention branch shown); when the code falls through to the else branch — any sequence parallel mode other than the supported Ulysses/flash-attn combination — it raises NotImplementedError explicitly, signalling planned-but-missing support.","triggerScenarios":"Enabling sequence parallelism with an attention implementation other than flash_attention_2 (e.g. sdpa/eager) or a CP mode the plugin does not implement, so control reaches the else branch in the patched attention forward.","commonSituations":"Setting compute_strategy: use_flash_attention off while cp_size > 1; mixing sequence parallel with sdpa or eager attention; config templates that enable ulysses regardless of attention backend.","solutions":["Set the attention implementation to flash_attention_2 when using ulysses sequence parallelism","Reduce/disable sequence parallelism (cp_size = 1) if you must use sdpa/eager","Track the upstream repo for additional SP modes; do not attempt to bypass the raise — other paths are genuinely unimplemented"],"exampleFix":"# before\ncompute_strategy:\n  cp_size: 4\nflash_attn: fa3  # or attention left as sdpa\n\n# after\ncompute_strategy:\n  cp_size: 4\n  use_flash_attention: true  # ulysses requires the flash-attn path","handlingStrategy":"validation","validationCode":"if cp_size > 1 and not config.get(\"use_flash_attention\", False):\n    raise ValueError(\"ulysses sequence parallel requires flash_attention_2; set it or use cp_size=1\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always pair sequence parallelism with flash-attention in configs","Add a config lint rule: cp_size>1 implies flash attention backend"],"tags":["not-implemented","sequence-parallel","attention","distributed"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}