{"record":{"id":"988d779d5929f8e5","repo":"labmlai/annotated_deep_learning_paper_implementations","slug":"head-size-d-k-too-large-for-flash-attention","errorCode":null,"errorMessage":"Head size {d_k} too large for flash attention","messagePattern":"Head size (.+?) too large for flash attention","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"labml_nn/neox/model.py","lineNumber":308,"sourceCode":"\n        # Reshape from `[batch_size, seq_len, n_heads, d_k] to `[batch_size, seq_len, n_hidden]`\n        output = output.reshape(*x.shape)\n\n        # Final linear layer\n        return self.output(output)\n\n    def compute_flash_attention(self, q: torch.Tensor, k: torch.Tensor, v: torch.Tensor):\n        # Stack them into shape `[batch_size, seq_len, 3, n_heads, d_k]`\n        qkv = torch.stack((q, k, v), dim=2)\n        d_k = qkv.shape[-1]\n        if d_k <= 32:\n            pad = 32 - d_k\n        elif d_k <= 64:\n            pad = 64 - d_k\n        elif d_k <= 128:\n            pad = 128 - d_k\n        else:\n            raise ValueError(f'Head size {d_k} too large for flash attention')\n\n        if pad > 0:\n            qkv = torch.cat((qkv, qkv.new_zeros(*qkv.shape[:-1], pad)), dim=-1)\n\n        output, _ = self.flash_attention(qkv, causal=True)\n        # The output is of shape `[batch_size, seq_len, n_heads, d_k + padding]`\n        output = output[:, :, :, :d_k]\n\n        return output\n\n    def compute_attention(self, q: torch.Tensor, k: torch.Tensor, v: torch.Tensor):\n        # Disable auto-casting to fp16 for attention computation\n        with autocast(enabled=False):\n            if q.dtype == torch.float16:\n                # Convert to fp32 if the current dtype is fp16\n                attn = torch.einsum('bihk,bjhk->bijh', q.float(), k.float())\n            else:\n                # Do not cast for bfloat","sourceCodeStart":290,"sourceCodeEnd":326,"githubUrl":"https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/33ab02281c2b928e6b32792909cc79cbdcfe1d6a/labml_nn/neox/model.py#L290-L326","documentation":"GPT-NeoX's compute_flash_attention pads the key/query head dimension d_k to the next supported size (32/64/128) before invoking the fused flash attention kernel. d_k above 128 exceeds the kernel's maximum supported head size, so the code raises ValueError instead of calling the kernel with an unsupported shape.","triggerScenarios":"Running NeoX model forward() with d_model // n_heads > 128 (e.g. d_model=12288 with 96 heads gives d_head=128 which is fine, but d_model=12288 with 48 heads gives 256 which raises); directly calling compute_flash_attention with an oversized qkv tensor.","commonSituations":"Scaling down n_heads when adapting NeoX configs; using configs copied from models that relied on PyTorch's built-in SDPA which supports larger head dims; mismatched d_model/n_heads after editing a YAML config.","solutions":["Change n_heads so that d_model // n_heads <= 128","Fall back to standard attention: replace compute_flash_attention with a plain softmax(QK^T/sqrt(d_k))V implementation for those layers","Validate the config at load time: assert d_model % n_heads == 0 and d_model // n_heads <= 128"],"exampleFix":"# before: d_model=8192, n_heads=32 -> d_k=256 -> raises\nmodel = NeoX(d_model=8192, n_heads=32)\n\n# after: d_k=64 -> OK\nmodel = NeoX(d_model=8192, n_heads=128)","handlingStrategy":"validation","validationCode":"d_model, n_heads = 8192, 128\nassert d_model % n_heads == 0 and d_model // n_heads <= 128, \\n    f'd_k={d_model // n_heads} exceeds flash-attention limit of 128'","typeGuard":"def neox_head_size_supported(d_model: int, n_heads: int) -> bool:\n    return d_model % n_heads == 0 and d_model // n_heads <= 128","tryCatchPattern":"try:\n    logits = model(idx, target)\nexcept ValueError as e:\n    if 'too large for flash attention' in str(e):\n        raise SystemExit('Fix NeoX config: d_model // n_heads must be <= 128')\n    raise","preventionTips":["Validate NeoX YAML configs (d_model, n_heads) in a pre-launch script","When porting configs, recompute head size rather than trusting the source repo","Keep a plain-attention fallback implementation available for odd configs"],"tags":["pytorch","gpt-neox","flash-attention","config"],"backgroundTag":"attention-head-dim-limit","analyzedSha":"33ab02281c2b928e6b32792909cc79cbdcfe1d6a","analyzedAt":"2026-08-25T10:30:27.743Z","schemaVersion":2},"datasetVersion":"2026-08-25T11:17:15.655Z"}