sgl-project/sglang · critical · NotImplementedError

DSV4 ragged verify does not support context parallel (CP); s

Error message

DSV4 ragged verify does not support context parallel (CP); set SGLANG_RAGGED_VERIFY_MODE off for CP runs.

What it means

The DSV4 ragged-verify layout (enabled when SGLANG_RAGGED_VERIFY_MODE is COMPACT) reorganizes speculative verify tokens but has no support for attention context parallelism across ranks. The backend raises during metadata resolution (init_forward_metadata_out_graph / _build_forward_metadata) rather than producing incorrect cross-rank attention.

Source

Thrown at python/sglang/srt/layers/attention/deepseek_v4_backend.py:642

        self._verify_mask = None
        self.cuda_graph_swa_out_cache_loc: Optional[torch.Tensor] = None

    def _move_to_device(self, x: List[int]) -> torch.Tensor:
        pin_tensor = torch.tensor(x, dtype=torch.int32, pin_memory=True)
        return pin_tensor.to(self.device, non_blocking=True)

    def _resolve_verify_layout(
        self,
        forward_batch: ForwardBatch,
        bs: int,
    ) -> Optional[RaggedVerifyLayout]:
        layout = resolve_ragged_verify_layout(forward_batch)
        if layout is None:
            return None
        if read_ragged_verify_mode() is not RaggedVerifyMode.COMPACT:
            return None
        if get_parallel().attn_cp_size > 1:
            raise NotImplementedError(
                "DSV4 ragged verify does not support context parallel (CP); "
                "set SGLANG_RAGGED_VERIFY_MODE off for CP runs."
            )
        if self.online_c128_mtp.enabled():
            raise NotImplementedError(
                "DSV4 ragged verify does not support online c128 MTP; "
                "set SGLANG_RAGGED_VERIFY_MODE off or disable online compress."
            )
        # Layout invariants (verify_lens >= 1, total == sum) are enforced in
        # RaggedVerifyLayout.__post_init__; don't re-check the device tensor
        # here -- that would D2H-sync the host-free verify prep path.
        layout = layout.padded_to_bucket(padded_bs=bs)
        return layout

    def _target_verify_graph_key(
        self,
        bs: int,
        ragged_layout: Optional[RaggedVerifyLayout],

View on GitHub (pinned to 0132848349)

Solutions

Unset/disable SGLANG_RAGGED_VERIFY_MODE (set it off) for CP runs
Disable attention context parallelism if ragged verify mode is required

Example fix

# before
SGLANG_RAGGED_VERIFY_MODE=compact sglang_launch ... --cp-size 4
# after
unset SGLANG_RAGGED_VERIFY_MODE; sglang_launch ... --cp-size 4

Defensive patterns

Strategy: validation

Validate before calling

import os
from sglang.srt.distributed import get_parallel
if get_parallel().attn_cp_size > 1 and os.environ.get("SGLANG_RAGGED_VERIFY_MODE", "").lower() == "compact":
    del os.environ["SGLANG_RAGGED_VERIFY_MODE"]  # incompatible with CP

Prevention

Centralize SGLANG_* env var review when changing parallelism settings
Run a short smoke launch when toggling CP size with experimental env vars set

When it happens

Trigger: Running with attn_cp_size > 1 while read_ragged_verify_mode() returns RaggedVerifyMode.COMPACT and a ragged verify layout resolves for the forward batch.

Common situations: Exporting SGLANG_RAGGED_VERIFY_MODE=compact for verify-path speed while enabling CP (e.g. --cp-size / attn CP) on a multi-GPU DeepSeek-V4 deployment; env vars left set from single-GPU experiments.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/58b82c7979e9032e. Report an issue: GitHub.