sgl-project/sglang · error · ValueError

The hpc_ops attention backend only supports the default soft

Error message

The hpc_ops attention backend only supports the default softmax scaling head_dim ** -0.5, got {layer.scaling}.

What it means

The HPC-Ops attention kernels hard-code the softmax scale to head_dim ** -0.5, so the backend validates that layer.scaling matches this default. If the model computes a non-default softmax scale (e.g. partial rotary embeddings with different scaling, or an explicit scaling override), the backend raises this ValueError because results would be silently incorrect.

Source

Thrown at python/sglang/srt/layers/attention/hpc_ops_backend.py:352

                metadata.cache_seqlens_int32
            )
        self.forward_metadata = metadata

    def get_cuda_graph_seq_len_fill_value(self) -> int:
        return 1

    def _check_layer_supported(self, layer: RadixAttention):
        if layer.sliding_window_size is not None and layer.sliding_window_size > -1:
            raise ValueError(
                "The hpc_ops attention backend does not support sliding window."
            )
        if layer.logit_cap > 0:
            raise ValueError(
                "The hpc_ops attention backend does not support logit cap."
            )
        # The HPC-Ops kernels hard-code the softmax scale to head_dim ** -0.5.
        if abs(layer.scaling - layer.head_dim**-0.5) > 1e-8:
            raise ValueError(
                "The hpc_ops attention backend only supports the default "
                f"softmax scaling head_dim ** -0.5, got {layer.scaling}."
            )

    def _paged_kv_buffers(self, layer: RadixAttention):
        k_cache, v_cache = self.token_to_kv_pool.get_kv_buffer(layer.layer_id)
        k_cache = k_cache.view(-1, self.page_size, layer.tp_k_head_num, layer.head_dim)
        v_cache = v_cache.view(-1, self.page_size, layer.tp_v_head_num, layer.head_dim)
        return k_cache, v_cache

    def _layer_kv_scales(self, layer: RadixAttention):
        """Per-tensor K/V scales as fp32 [1] tensors (ones when absent)."""
        k_scale = (
            layer.k_scale.reshape(1).float()
            if layer.k_scale is not None
            else self._ones_scale
        )
        v_scale = (

View on GitHub (pinned to 0132848349)

Solutions

  1. Use a different attention backend (flashinfer, fa3, triton) that respects the model's softmax scaling
  2. Inspect the model's attention module to see how scaling is computed; if it should be the default, fix the model code to use head_dim ** -0.5
  3. Do not force hpc_ops for this model; let sglang auto-select the backend

Example fix

# before
self.attn = RadixAttention(..., scaling=custom_scale)  # with --attention-backend hpc_ops
# after
self.attn = RadixAttention(..., scaling=head_dim ** -0.5)  # or switch backend
Defensive patterns

Strategy: validation

Validate before calling

def scaling_ok(layer) -> bool:
    return abs(layer.scaling - layer.head_dim ** -0.5) <= 1e-8

assert scaling_ok(attn_layer), f'non-default scaling {attn_layer.scaling}'

Type guard

def has_default_softmax_scale(layer: RadixAttention) -> bool:
    return abs(layer.scaling - layer.head_dim ** -0.5) <= 1e-8

Try / catch

try:
    backend.forward_decode(...)
except ValueError as e:
    if 'softmax scaling' in str(e):
        switch_attention_backend('flashinfer')
    else:
        raise

Prevention

When it happens

Trigger: Calling forward_extend/forward_decode with --attention-backend hpc_ops on a model whose RadixAttention layers pass a custom scaling value different from head_dim ** -0.5 (within 1e-8 tolerance).

Common situations: Models with unusual head_dim/scaling configurations (some Qwen/Yi variants or custom models), or architectures that set layer.scaling explicitly in their attention module instead of using the default.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/d1bbc609eda1ec66. Report an issue: GitHub.