sgl-project/sglang · error · ValueError

DSpark with dp attention does not support context parallel (

Error message

DSpark with dp attention does not support context parallel (attn_cp_size={}).

What it means

DSpark with dp attention cannot run together with attention context parallelism. The hook rejects attn_cp_size > 1 under DP attention because CP-sharded attention is incompatible with the DSpark DP verification path.

Source

Thrown at python/sglang/srt/arg_groups/speculative_hook.py:378

            raise ValueError(
                "DSpark with dp attention supports moe_a2a_backend 'none' "
                "(built-in TP MoE) or 'megamoe', got "
                f"{cfg.moe_a2a_backend!r}."
            )
        if not _is_npu and cfg.moe_a2a_backend != "none":
            from sglang.srt.speculative.ragged_verify import (
                RaggedVerifyMode,
                read_ragged_verify_mode,
            )

            if read_ragged_verify_mode() is not RaggedVerifyMode.STATIC:
                raise ValueError(
                    "DSpark with dp attention + "
                    f"moe_a2a_backend={cfg.moe_a2a_backend!r} requires "
                    "SGLANG_RAGGED_VERIFY_MODE=static."
                )
        if cfg.attn_cp_size > 1:
            raise ValueError(
                "DSpark with dp attention does not support context parallel "
                f"(attn_cp_size={cfg.attn_cp_size})."
            )
        if (
            not _is_npu
            and cfg.speculative_moe_a2a_backend is not None
            and cfg.speculative_moe_a2a_backend != cfg.moe_a2a_backend
        ):
            raise ValueError(
                "DSpark ignores --speculative-moe-a2a-backend; with dp attention it "
                f"must match the target moe_a2a_backend={cfg.moe_a2a_backend!r} "
                f"(got {cfg.speculative_moe_a2a_backend!r})."
            )

    if cfg.pp_size != 1:
        raise ValueError(
            "Currently DSpark speculative decoding only supports pp_size == 1."
        )

View on GitHub (pinned to 0132848349)

Solutions

Set --attn-cp-size 1 (or drop the flag) when using DSpark + dp attention
Drop dp attention if context parallelism is required
Use a different speculative algorithm that supports CP

Example fix

# before
--speculative-algorithm DSPARK --enable-dp-attention --dp-size 4 --attn-cp-size 2
# after
--speculative-algorithm DSPARK --enable-dp-attention --dp-size 4

Defensive patterns

Strategy: validation

Validate before calling

if args.enable_dp_attention and args.dp_size > 1 and args.attn_cp_size > 1:
    raise SystemExit('DSPARK + dp attention does not support attn_cp_size > 1')

Prevention

Long-context CP configs and DSpark DP attention are mutually exclusive; keep separate presets

When it happens

Trigger: Launching DSpark with --enable-dp-attention, --dp-size>1, and --attn-cp-size (attn_cp_size) greater than 1.

Common situations: Long-context deployments (DeepSeek V4-style) that enable attention CP for ultra-long sequences also enable DP attention; adding DSpark breaks the combo.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/65f6e7ea18d78a6d. Report an issue: GitHub.