{"record":{"id":"3871c357d26fa6af","repo":"sgl-project/sglang","slug":"cannot-determine-attention-scale-for-type-inner","errorCode":null,"errorMessage":"Cannot determine attention scale for {type(inner).__name__}","messagePattern":"Cannot determine attention scale for (.+?)","errorType":"error_code","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/hardware_backend/mlx/kv_cache/attention_wrapper.py","lineNumber":207,"sourceCode":"    the trailing window of the cached keys only, which is numerically\n    identical to a rotating cache.  Both cache kinds keep KV in temporal\n    order and report absolute offsets, so the same trailing-window slice\n    works whether the cache holds full history or only the window.\n    \"\"\"\n\n    def __init__(\n        self, inner: nn.Module, layer_idx: int, window_size: int | None = None\n    ):\n        super().__init__()\n        object.__setattr__(self, \"_inner\", inner)\n        object.__setattr__(self, \"_layer_idx\", layer_idx)\n        object.__setattr__(self, \"_window_size\", window_size)\n        # Resolved once at patch time (weights are loaded before patching and\n        # the inner module is never swapped afterwards), keeping the decode\n        # hot path free of attribute scans and failing fast on a bad module.\n        scale = get_attention_scale(inner)\n        if scale is None:\n            raise RuntimeError(\n                f\"Cannot determine attention scale for {type(inner).__name__}\"\n            )\n        n_heads = get_num_heads(inner)\n        n_kv_heads = get_num_kv_heads(inner)\n        if n_heads is None or n_kv_heads is None:\n            raise RuntimeError(\n                f\"Cannot determine attention head counts for {type(inner).__name__}\"\n            )\n        object.__setattr__(self, \"_scale\", scale)\n        object.__setattr__(self, \"_n_heads\", n_heads)\n        object.__setattr__(self, \"_n_kv_heads\", n_kv_heads)\n        # None for modules that expose head_dim only through a projection\n        # shape; _batched_decode falls back to the runtime K shape.\n        object.__setattr__(self, \"_head_dim\", get_head_dim(inner))\n        object.__setattr__(self, \"_has_q_norm\", hasattr(inner, \"q_norm\"))\n        object.__setattr__(self, \"_has_k_norm\", hasattr(inner, \"k_norm\"))\n        # Only pass sinks when the module has them: the kwarg requires a\n        # recent mlx and must not constrain models without sinks.","sourceCodeStart":189,"sourceCodeEnd":225,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/hardware_backend/mlx/kv_cache/attention_wrapper.py#L189-L225","documentation":"When patching an mlx_lm attention module, the wrapper resolves the attention softmax scale once via `get_attention_scale(inner)`. If the inner module exposes none of the recognized attributes (e.g. `scale`, `head_dim`), the helper returns None and the wrapper raises at patch time — deliberately, so the decode hot path never has to guess a scale.","triggerScenarios":"Wrapping an attention module whose type doesn't expose a known scale attribute — a new/custom attention implementation, an upstream mlx_lm rename, or a model class not yet supported by the MLX backend.","commonSituations":"Loading a brand-new mlx_lm model architecture; upgrading mlx_lm so attention internals are renamed; using a custom attention subclass with non-standard attribute names.","solutions":["Extend `get_attention_scale` to recognize the new module's scale attribute (or expose `scale`/`head_dim` on your custom module).","Pin mlx_lm to a supported version for your SGLang release.","Skip MLX KV-cache patching for unsupported architectures (fall back to default path)."],"exampleFix":"# before\nclass MyAttention(nn.Module):\n    def __init__(self, d):\n        self.softmax_scale = d ** -0.5  # unrecognized name\n\n# after\nclass MyAttention(nn.Module):\n    def __init__(self, d):\n        self.scale = d ** -0.5  # recognized by get_attention_scale","handlingStrategy":"validation","validationCode":"from sglang.srt.hardware_backend.mlx.kv_cache.attention_wrapper import get_attention_scale\nif get_attention_scale(module) is None:\n    module.scale = module.head_dim ** -0.5  # or skip patching","typeGuard":"def is_patchable_attention(module) -> bool:\n    return get_attention_scale(module) is not None","tryCatchPattern":null,"preventionTips":["Pin mlx_lm versions validated for your model set.","Smoke-test wrapper patching per new model architecture.","Expose `scale` on custom attention modules."],"tags":["mlx","attention","model-compatibility","patching"],"backgroundTag":"unsupported-model-architecture","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}