{"record":{"id":"d1bbc609eda1ec66","repo":"sgl-project/sglang","slug":"the-hpc-ops-attention-backend-only-supports-the-de","errorCode":null,"errorMessage":"The hpc_ops attention backend only supports the default softmax scaling head_dim ** -0.5, got {layer.scaling}.","messagePattern":"The hpc_ops attention backend only supports the default softmax scaling head_dim \\*\\* -0\\.5, got (.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/attention/hpc_ops_backend.py","lineNumber":352,"sourceCode":"                metadata.cache_seqlens_int32\n            )\n        self.forward_metadata = metadata\n\n    def get_cuda_graph_seq_len_fill_value(self) -> int:\n        return 1\n\n    def _check_layer_supported(self, layer: RadixAttention):\n        if layer.sliding_window_size is not None and layer.sliding_window_size > -1:\n            raise ValueError(\n                \"The hpc_ops attention backend does not support sliding window.\"\n            )\n        if layer.logit_cap > 0:\n            raise ValueError(\n                \"The hpc_ops attention backend does not support logit cap.\"\n            )\n        # The HPC-Ops kernels hard-code the softmax scale to head_dim ** -0.5.\n        if abs(layer.scaling - layer.head_dim**-0.5) > 1e-8:\n            raise ValueError(\n                \"The hpc_ops attention backend only supports the default \"\n                f\"softmax scaling head_dim ** -0.5, got {layer.scaling}.\"\n            )\n\n    def _paged_kv_buffers(self, layer: RadixAttention):\n        k_cache, v_cache = self.token_to_kv_pool.get_kv_buffer(layer.layer_id)\n        k_cache = k_cache.view(-1, self.page_size, layer.tp_k_head_num, layer.head_dim)\n        v_cache = v_cache.view(-1, self.page_size, layer.tp_v_head_num, layer.head_dim)\n        return k_cache, v_cache\n\n    def _layer_kv_scales(self, layer: RadixAttention):\n        \"\"\"Per-tensor K/V scales as fp32 [1] tensors (ones when absent).\"\"\"\n        k_scale = (\n            layer.k_scale.reshape(1).float()\n            if layer.k_scale is not None\n            else self._ones_scale\n        )\n        v_scale = (","sourceCodeStart":334,"sourceCodeEnd":370,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/attention/hpc_ops_backend.py#L334-L370","documentation":"The HPC-Ops attention kernels hard-code the softmax scale to head_dim ** -0.5, so the backend validates that layer.scaling matches this default. If the model computes a non-default softmax scale (e.g. partial rotary embeddings with different scaling, or an explicit scaling override), the backend raises this ValueError because results would be silently incorrect.","triggerScenarios":"Calling forward_extend/forward_decode with --attention-backend hpc_ops on a model whose RadixAttention layers pass a custom scaling value different from head_dim ** -0.5 (within 1e-8 tolerance).","commonSituations":"Models with unusual head_dim/scaling configurations (some Qwen/Yi variants or custom models), or architectures that set layer.scaling explicitly in their attention module instead of using the default.","solutions":["Use a different attention backend (flashinfer, fa3, triton) that respects the model's softmax scaling","Inspect the model's attention module to see how scaling is computed; if it should be the default, fix the model code to use head_dim ** -0.5","Do not force hpc_ops for this model; let sglang auto-select the backend"],"exampleFix":"# before\nself.attn = RadixAttention(..., scaling=custom_scale)  # with --attention-backend hpc_ops\n# after\nself.attn = RadixAttention(..., scaling=head_dim ** -0.5)  # or switch backend","handlingStrategy":"validation","validationCode":"def scaling_ok(layer) -> bool:\n    return abs(layer.scaling - layer.head_dim ** -0.5) <= 1e-8\n\nassert scaling_ok(attn_layer), f'non-default scaling {attn_layer.scaling}'","typeGuard":"def has_default_softmax_scale(layer: RadixAttention) -> bool:\n    return abs(layer.scaling - layer.head_dim ** -0.5) <= 1e-8","tryCatchPattern":"try:\n    backend.forward_decode(...)\nexcept ValueError as e:\n    if 'softmax scaling' in str(e):\n        switch_attention_backend('flashinfer')\n    else:\n        raise","preventionTips":["Log layer.scaling and head_dim at model load when using hpc_ops","Unit-test model attention configs against backend constraints","Let sglang pick the backend for exotic architectures"],"tags":["attention-backend","hpc-ops","softmax-scaling","head-dim","sglang"],"backgroundTag":"unsupported-attention-backend-feature","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}