sgl-project/sglang · error · NotImplementedError

LingBot causal sequence sharding currently requires kv_cache

Error message

LingBot causal sequence sharding currently requires kv_cache-backed inference.

What it means

LingBot's causal sequence sharding relies on a paged KV cache to gather full-sequence history on each shard. When sequence sharding is enabled (forward_batch.enable_sequence_shard with ulysses degree > 1) but kv_cache is None, the code paths can't reconstruct the causal context, so it raises NotImplementedError.

Source

Thrown at python/sglang/multimodal_gen/runtime/models/dits/lingbot_world.py:267

            )
            roped_query = roped_query.type_as(v)
            roped_key = roped_key.type_as(v)
        else:
            roped_query = _apply_rotary_emb(q, cos, sin, is_neox_style=False).type_as(v)
            roped_key = _apply_rotary_emb(k, cos, sin, is_neox_style=False).type_as(v)
        forward_batch = get_forward_context().forward_batch
        seq_splits = None
        uniform_seq_splits = False
        sequence_shard_enabled = (
            kv_cache is not None
            and forward_batch is not None
            and getattr(forward_batch, "enable_sequence_shard", False)
            and get_ulysses_parallel_world_size() > 1
        )

        if kv_cache is None:
            if sequence_shard_enabled:
                raise NotImplementedError(
                    "LingBot causal sequence sharding currently requires kv_cache-backed inference."
                )
            return super().forward(
                q,
                k,
                v,
                (cos, sin),
                block_mask,
                kv_cache,
                current_start,
                cache_start,
            )

        if sequence_shard_enabled:
            seq_splits = getattr(forward_batch, "sequence_shard_splits", None)
            if seq_splits is None:
                raise ValueError(
                    "LingBot causal sequence sharding requires forward_batch.sequence_shard_splits."

View on GitHub (pinned to 0132848349)

Solutions

  1. Run through the serving/scheduler path so forward_batch carries a kv_cache
  2. Disable sequence sharding for eager/offline runs (leave enable_sequence_shard unset/False)
  3. Extend the model with a non-cache fallback if you need sharded eager execution (code change)

Example fix

# before
out = attn(q, k, v, forward_batch=batch)  # batch.enable_sequence_shard=True, no kv_cache

# after
batch.enable_sequence_shard = False       # eager / cacheless run
out = attn(q, k, v, forward_batch=batch)
Defensive patterns

Strategy: type-guard

Validate before calling

shard_ok = (forward_batch is not None\n              and getattr(forward_batch, 'enable_sequence_shard', False)\n              and get_ulysses_parallel_world_size() > 1\n              and forward_batch.token_to_kv_pool is not None)\nif not shard_ok:\n    forward_batch.enable_sequence_shard = False

Type guard

def has_kv_cache(forward_batch) -> bool:\n    return getattr(forward_batch, 'token_to_kv_pool', None) is not None

Try / catch

try:\n    out = attn(q, k, v, forward_batch=batch, kv_cache=cache)\nexcept NotImplementedError:\n    batch.enable_sequence_shard = False\n    out = attn(q, k, v, forward_batch=batch)

Prevention

When it happens

Trigger: Calling forward with enable_sequence_shard=True, ulysses world size > 1, and kv_cache=None (e.g. a pure eager forward with no radix cache attached, as in unit tests or offline single-step runs).

Common situations: Running the block standalone in tests without the scheduler's KV cache; a batch type that forgot to attach cache tensors; trying the sharded path in a non-serving script.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/70c5603507734e5b. Report an issue: GitHub.