sgl-project/sglang · error · ValueError

ReplaySSM cache length must be at least 1.

Error message

ReplaySSM cache length must be at least 1.

What it means

The ReplaySSM caches (d_cache, k_cache, g_cache) are ring buffers along a length axis; the kernel reads d_cache.size(2) as the cache length and requires it >= 1. A zero-length cache (empty third dim) has no slot to write flushed rows into, so the guard raises.

Source

Thrown at python/sglang/kernels/ops/attention/helion/kda_replayssm.py:736

        initial_state,
        out,
        ssm_state_indices,
    )

    if write_pos.ndim != 1 or write_pos.dtype is not torch.int32:
        raise ValueError("`write_pos` must be a 1D int32 tensor.")
    if write_pos.shape != (batch,):
        raise ValueError(f"`write_pos` must have shape {(batch,)}.")
    if force_flush is not None and (
        force_flush.ndim != 1
        or force_flush.dtype is not torch.int32
        or force_flush.shape != (batch,)
    ):
        raise ValueError("`force_flush` must be a length-B int32 tensor or None.")

    cache_length = d_cache.size(2)
    if cache_length < 1:
        raise ValueError("ReplaySSM cache length must be at least 1.")
    if d_cache.shape[1:] != (num_v_heads, cache_length, value_dim):
        raise ValueError("`d_cache` must have shape [slots, HV, L, V].")
    if k_cache.shape[1:] != (num_q_heads, cache_length, key_dim):
        raise ValueError("`k_cache` must have shape [slots, H, L, K].")
    if g_cache.shape[1:] != (num_v_heads, cache_length, key_dim):
        raise ValueError("`g_cache` must have shape [slots, HV, L, K].")
    if g_cache.dtype is not torch.float32:
        raise ValueError("`g_cache` must have dtype torch.float32.")

    device = mixed_qkv.device
    if any(
        tensor.device != device for tensor in (d_cache, k_cache, g_cache, write_pos)
    ):
        raise ValueError("ReplaySSM inputs must be on the same device.")
    if force_flush is not None and force_flush.device != device:
        raise ValueError("`force_flush` must be on the same device as the inputs.")

    cache_block = helion.next_power_of_2(max(16, cache_length))

View on GitHub (pinned to 0132848349)

Solutions

1. Allocate the cache with a positive length, at least 1: d_cache = torch.empty(slots, HV, L, V) with L >= 1
2. If replay is disabled, use the non-replayssm decode entry point instead of a zero-length cache
3. Validate cache config at startup: assert cache_len >= 1 when replayssm is enabled

Example fix

// before
d_cache = torch.empty(slots, HV, 0, V, device=dev, dtype=dt)  # cache_len=0
// after
d_cache = torch.empty(slots, HV, max(1, cache_len), V, device=dev, dtype=dt)

assert d_cache.size(2) >= 1, f"cache length {d_cache.size(2)} must be >= 1"
# or at allocation:
L = max(1, cfg.replay_cache_len)
d_cache = torch.empty(slots, HV, L, V, device=dev, dtype=dt)

def valid_d_cache(t: torch.Tensor) -> bool:
    return t.size(2) >= 1

When it happens

Trigger: Allocating d_cache as torch.empty([slots, HV, 0, V]) — e.g. a pool sized from a config field that evaluated to 0 (cache_len=0 or replay window misconfigured).

Common situations: Configuring replay/window length to 0 when the feature is meant to be disabled instead of passing None; allocating caches from a shape tuple with a typo; a model config default of 0 for a new field feeding cache allocation.

Related errors

• KDA `a` must be a contiguous 2D or 3D tensor.
• KDA `dt_bias` must be a contiguous 1D or 2D tensor.
• `write_pos` must be a 1D int32 tensor.
• `write_pos` must have shape {(batch,)}.
• `force_flush` must be a length-B int32 tensor or None.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/1c1e6f4c9a927948. Report an issue: GitHub.