sgl-project/sglang · error · AssertionError

online c128 does not support MTP

Error message

online c128 does not support MTP

What it means

Online c128 compression (compress_ratio=128 with ONLINE_C128 enabled) keeps a single (max, sum, kv) state per index instead of a speculative ring buffer, so it cannot accommodate speculative decoding (MTP). When is_speculative is set and the SGLANG_EXPERIMENTAL_ONLINE_C128_MTP escape hatch is unset, this AssertionError fires from get_compress_state_ring_size.

Source

Thrown at python/sglang/srt/mem_cache/deepseek_v4_memory_pool.py:43

from sglang.srt.utils import ceil_div, is_hip

logger = logging.getLogger(__name__)

_is_hip = is_hip()

ONLINE_C128 = not _is_hip and envs.SGLANG_OPT_USE_ONLINE_COMPRESS.get()


def get_compress_state_ring_size(
    compress_ratio: int, is_speculative: bool = False
) -> int:
    assert compress_ratio in [4, 128], f"Unsupported {compress_ratio = }"
    # Online c128 keeps a single (max, sum, kv) state per index instead of a
    # 128-slot ring buffer of raw tokens, so ring_size collapses to 1. Online
    # is incompatible with speculative decode for now.
    if compress_ratio == 128 and ONLINE_C128:
        if is_speculative and not envs.SGLANG_EXPERIMENTAL_ONLINE_C128_MTP.get():
            raise AssertionError("online c128 does not support MTP")
        return 1
    if is_speculative:
        return 16 if compress_ratio == 4 else 256
    else:
        return 8 if compress_ratio == 4 else 128


def get_compress_state_write_pad(compress_ratio: int, ring_size: int) -> int:
    """Largest draft-token count this ring can serve; mirrors `mtp_pad` in `c_plan.cuh`
    (the bound is derived there). Zero for a non-speculative ring, which is exactly one
    window wide."""
    window_size = compress_ratio * (2 if compress_ratio == 4 else 1)
    return ring_size - window_size + 2 if ring_size > window_size else 0


class DeepSeekV4SingleKVPool(KVCache):
    def __init__(
        self,

View on GitHub (pinned to 0132848349)

Solutions

1. Disable speculative decoding (remove --speculative-algorithm / MTP flags) for online c128 models
2. Use compress_ratio=4 instead of 128 if MTP is required
3. If you accept the experimental risk, set SGLANG_EXPERIMENTAL_ONLINE_C128_MTP env var to enable the unsupported combination

Example fix

# before
--model deepseek-ai/DeepSeek-V4 --speculative-algorithm EAGLE ...
# after
--model deepseek-ai/DeepSeek-V4   # no speculative flags with online c128

from sglang.srt.mem_cache.deepseek_v4_memory_pool import ONLINE_C128
if ONLINE_C128 and server_args.speculative_algorithm and compress_ratio == 128:
    assert envs.SGLANG_EXPERIMENTAL_ONLINE_C128_MTP.get(), "online c128 + MTP unsupported"

def online_c128_mtp_ok(is_speculative: bool) -> bool:
    return (not is_speculative) or bool(envs.SGLANG_EXPERIMENTAL_ONLINE_C128_MTP.get())

When it happens

Trigger: Launching a DeepSeek-V4 c128 model with ONLINE_C128 enabled together with speculative decoding (--speculative-algorithm / MTP) and without SGLANG_EXPERIMENTAL_ONLINE_C128_MTP=...; called during memory pool init (get_ring_size/__init__).

Common situations: Enabling MTP speculative decoding on a DSv4 online-c128 deployment for throughput; upgrading SGLang where online c128 became the default; test code that exercises ring capacity with speculative flags set.

Related errors

• DSV4 draft state transfer expects SWA-only NextN layers
• DSV4 ragged verify does not support context parallel (CP); s
• DSV4 ragged verify does not support online c128 MTP; set SGL
• num_nextn_predict_layers is not in the config
• num nextn_predict_layers is not in the config

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/5026c062eb5a3b3d. Report an issue: GitHub.