{"record":{"id":"6ca03dae2b904f19","repo":"dotnet/machinelearning","slug":"embeddingdim-must-be-divisible-by-numheads","errorCode":null,"errorMessage":"EmbeddingDim must be divisible by NumHeads","messagePattern":"EmbeddingDim must be divisible by NumHeads","errorType":"exception","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.TorchSharp/NasBert/Modules/MultiHeadAttention.cs","lineNumber":77,"sourceCode":"            bool bias = true,\n            bool addBiasKv = false,\n            bool addZeroAttention = false,\n            bool selfAttention = false,\n            bool encoderDecoderAttention = false)\n            : base(nameof(MultiHeadAttention))\n        {\n            _embeddingDim = embeddingDim;\n            _kDim = kDim ?? embeddingDim;\n            _vDim = vDim ?? embeddingDim;\n            _qkvSameDim = (_kDim == _embeddingDim) && (_vDim == _embeddingDim);\n\n            _numHeads = numHeads;\n            _dropout = dropout;\n            _headDim = _embeddingDim / _numHeads;\n            _scaling = Math.Pow(_headDim, -0.5);\n            if (_headDim * _numHeads != _embeddingDim)\n            {\n                throw new ArgumentException(\"EmbeddingDim must be divisible by NumHeads\");\n            }\n\n            _selfAttention = selfAttention;\n            _encoderDecoderAttention = encoderDecoderAttention;\n            if (_selfAttention && !_qkvSameDim)\n            {\n                throw new ArgumentException(\"Self-attention requires query, key and value to be of the same size\");\n            }\n\n            _addBiasProj = bias;\n            _addBiasKv = addBiasKv;\n            _addZeroAttention = addZeroAttention;\n\n            QProjection = torch.nn.Linear(_embeddingDim, _embeddingDim, _addBiasProj);\n            KProjection = torch.nn.Linear(_kDim, _embeddingDim, _addBiasProj);\n            VProjection = torch.nn.Linear(_vDim, _embeddingDim, _addBiasProj);\n\n            if (_addBiasKv)","sourceCodeStart":59,"sourceCodeEnd":95,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.TorchSharp/NasBert/Modules/MultiHeadAttention.cs#L59-L95","documentation":"MultiHeadAttention splits the embedding dimension across attention heads (headDim = embeddingDim / numHeads); if headDim * numHeads != embeddingDim, the heads cannot cover the full embedding dimension and the module throws ArgumentException. This is a constructor-time invariant check that the model dimensions are internally consistent.","triggerScenarios":"Calling the MultiHeadAttention constructor with embeddingDim that is not an exact multiple of numHeads, e.g. embeddingDim=100, numHeads=8, or a mismatched options object where hiddenSize and numAttentionHeads are configured independently.","commonSituations":"Hand-editing model architecture options (e.g., shrinking hiddenSize for a smaller model without adjusting head count), porting config JSON from another model whose dims differ, or computing numHeads dynamically with a rounding error.","solutions":["Choose numHeads that divides embeddingDim evenly (e.g., embeddingDim=768 with numHeads=12 gives headDim=64).","Validate before construction: `if (embeddingDim % numHeads != 0) throw ...` in your config-loading code.","Derive numHeads from embeddingDim programmatically (e.g., fixed headDim target) instead of hard-coding both.","Wrap the constructor call in try-catch on ArgumentException to translate it into a config validation error."],"exampleFix":"// before\nnew MultiHeadAttention(embeddingDim: 100, numHeads: 8); // throws\n\n// after\nnew MultiHeadAttention(embeddingDim: 768, numHeads: 12); // 768 % 12 == 0","handlingStrategy":"validation","validationCode":"if (embeddingDim % numHeads != 0)\n    throw new ArgumentException($\"embeddingDim ({embeddingDim}) must be divisible by numHeads ({numHeads}).\");","typeGuard":null,"tryCatchPattern":"try { var attn = new MultiHeadAttention(...); }\ncatch (ArgumentException ex) { log.LogError(ex, \"Invalid attention dimensions\"); throw new ModelConfigException(...); }","preventionTips":["Standard head counts: 768/12, 1024/16.","Derive numHeads from embeddingDim with a fixed headDim in config code.","Validate all architecture dims together when loading model options."],"tags":["csharp","attention","dimensions","argument-exception"],"backgroundTag":"invalid-argument-value","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}