sgl-project/sglang · error · NotImplementedError

trtllm_mla does not forward the cyclic DCP metadata to its d

Error message

trtllm_mla does not forward the cyclic DCP metadata to its decode kernel and returns no rank-local LSE for the cross-rank merge; select cutedsl_mla or tokenspeed_mla for a DCP target-verify run

What it means

trtllm_mla's decode path does not accept the cyclic-context-parallel (DCP) metadata arguments, and it never returns the rank-local LSE needed for the cross-rank merge in DCP target-verify runs. When forward_extend/forward_decode is invoked with cp_world > 1 or return_lse=True, the backend raises NotImplementedError rather than silently producing wrong results. The docstring states only DCP-capable subclasses (cutedsl_mla, tokenspeed_mla) can serve these calls.

Source

Thrown at python/sglang/srt/layers/attention/trtllm_mla_backend.py:859

        kv_cache: torch.Tensor,
        block_tables: torch.Tensor,
        seq_lens: torch.Tensor,
        max_seq_len: int,
        layer: RadixAttention,
        *,
        causal_seqs: Optional[torch.Tensor] = None,
        cp_world: int = 1,
        cp_rank: int = 0,
        return_lse: bool = False,
    ) -> torch.Tensor:
        """Hook for subclasses to swap the decode/spec-verify kernel.

        The DCP arguments belong to the hook contract because forward_extend
        passes them on the DCP target-verify path. This implementation does not
        forward them to the kernel and returns no LSE, so only the DCP-capable
        subclasses serve them."""
        if cp_world > 1 or return_lse:
            raise NotImplementedError(
                "trtllm_mla does not forward the cyclic DCP metadata to its "
                "decode kernel and returns no rank-local LSE for the cross-rank "
                "merge; select cutedsl_mla or tokenspeed_mla for a DCP "
                "target-verify run"
            )

        # Scale computation for TRTLLM MLA kernel BMM1 operation:
        # The final BMM1 scale is computed as: q_scale * k_scale * softmax_scale
        # Scale components:
        # - q_scale: Query scaling factor (set to 1.0 for both FP16/FP8 paths)
        # - k_scale: Key scaling factor from model checkpoint. Only applied when KV cache
        #   stores FP8-quantized values, to compensate for the quantization scaling.
        #   For BF16/FP16 KV cache, k_scale must be 1.0 since values are unscaled.
        # - softmax_scale: Attention softmax scaling = 1/sqrt(head_dim), pre-computed as layer.scaling
        bmm1_scale = self._compute_decode_bmm1_scale(layer)
        seq_lens_i32 = (
            seq_lens if seq_lens.dtype == torch.int32 else seq_lens.to(torch.int32)
        )

View on GitHub (pinned to 0132848349)

Solutions

Switch the MLA attention backend: use --attention-backend cutedsl_mla or tokenspeed_mla for DCP target-verify workloads.
Disable context parallelism (set cp_size=1) and do not request return_lse if you must keep trtllm_mla.
If you need trtllm_mla under DCP, implement forwarding of the cyclic DCP metadata into the decode kernel plus rank-local LSE output in a subclass and override _run_decode_kernel.

Example fix

# before
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3 --cp-size 2 --attention-backend trtllm_mla
# after
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3 --cp-size 2 --attention-backend cutedsl_mla

Defensive patterns

Strategy: validation

Validate before calling

from sglang.srt.server_args import ServerArgs
DCP_MLA_BACKENDS = {"cutedsl_mla", "tokenspeed_mla"}
args = parse_server_args()
if getattr(args, "cp_size", 1) > 1 and args.attention_backend not in DCP_MLA_BACKENDS:
    raise SystemExit(f"cp_size>1 requires one of {DCP_MLA_BACKENDS}, got {args.attention_backend}")

Try / catch

try:
    out = attn_backend.forward_decode(...)
except NotImplementedError as e:
    if "DCP" in str(e) or "return_lse" in str(e):
        raise SystemExit("switch to cutedsl_mla/tokenspeed_mla for DCP runs")
    raise

Prevention

Pin the attention backend explicitly instead of relying on defaults when using --cp-size.
Keep a compatibility matrix of (cp_size, speculative mode) vs supported MLA backends.
Fail fast at startup: validate server args before model load rather than at the first decode step.

When it happens

Trigger: Running DeepSeek-style MTP/DCP target-verify (--cp-size > 1, i.e. cp_world > 1) or requesting return_lse=True while the attention backend is set to trtllm_mla; _run_decode_kernel checks these hook-contract arguments and raises immediately.

Common situations: Enabling context parallelism or speculative target-verify on a multi-GPU node with --attention-backend trtllm_mla; mixing DCP configurations copied from a cutedsl_mla/tokenspeed_mla deployment into a trtllm_mla run.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/a2ea5f67a54f0d90. Report an issue: GitHub.