{"record":{"id":"a0274ac5e521dbda","repo":"dotnet/machinelearning","slug":"self-attention-requires-query-key-and-value-to-be","errorCode":null,"errorMessage":"Self-attention requires query, key and value to be of the same size","messagePattern":"Self-attention requires query, key and value to be of the same size","errorType":"exception","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.TorchSharp/NasBert/Modules/MultiHeadAttention.cs","lineNumber":84,"sourceCode":"            _embeddingDim = embeddingDim;\n            _kDim = kDim ?? embeddingDim;\n            _vDim = vDim ?? embeddingDim;\n            _qkvSameDim = (_kDim == _embeddingDim) && (_vDim == _embeddingDim);\n\n            _numHeads = numHeads;\n            _dropout = dropout;\n            _headDim = _embeddingDim / _numHeads;\n            _scaling = Math.Pow(_headDim, -0.5);\n            if (_headDim * _numHeads != _embeddingDim)\n            {\n                throw new ArgumentException(\"EmbeddingDim must be divisible by NumHeads\");\n            }\n\n            _selfAttention = selfAttention;\n            _encoderDecoderAttention = encoderDecoderAttention;\n            if (_selfAttention && !_qkvSameDim)\n            {\n                throw new ArgumentException(\"Self-attention requires query, key and value to be of the same size\");\n            }\n\n            _addBiasProj = bias;\n            _addBiasKv = addBiasKv;\n            _addZeroAttention = addZeroAttention;\n\n            QProjection = torch.nn.Linear(_embeddingDim, _embeddingDim, _addBiasProj);\n            KProjection = torch.nn.Linear(_kDim, _embeddingDim, _addBiasProj);\n            VProjection = torch.nn.Linear(_vDim, _embeddingDim, _addBiasProj);\n\n            if (_addBiasKv)\n            {\n                KBias = torch.zeros(1, 1, _embeddingDim).AsParameter();\n                VBias = torch.zeros(1, 1, _embeddingDim).AsParameter();\n            }\n\n            OutProjLinear = torch.nn.Linear(_embeddingDim, _embeddingDim, _addBiasProj);\n            DropoutLayer = torch.nn.Dropout(_dropout);","sourceCodeStart":66,"sourceCodeEnd":102,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.TorchSharp/NasBert/Modules/MultiHeadAttention.cs#L66-L102","documentation":"In self-attention the same tensor serves as query, key, and value, so their embedding dimensions must be equal; MultiHeadAttention throws ArgumentException when selfAttention=true but the supplied q/k/v dimensions differ (_qkvSameDim is false). This prevents constructing a module whose weights could never be shared/applied consistently.","triggerScenarios":"Calling the MultiHeadAttention constructor with selfAttention: true while passing different embeddingDim, keyDim, and valueDim values (e.g., embeddingDim=768, keyDim=512).","commonSituations":"Reusing a cross-attention configuration (different q/k/v sizes) in a self-attention slot, or misreading constructor parameter order and passing keyDim/valueDim where embeddingDim was expected.","solutions":["For self-attention, pass embeddingDim, keyDim, and valueDim as the same value (commonly all equal to hiddenSize).","Set selfAttention: false if you actually need cross-attention with differing q/k/v dims (as in encoder-decoder attention).","Validate dims before construction: `if (selfAttention && !(embeddingDim == keyDim && keyDim == valueDim)) throw ...`.","Wrap the constructor call in try-catch on ArgumentException to surface a configuration error."],"exampleFix":"// before\nnew MultiHeadAttention(embeddingDim: 768, keyDim: 512, valueDim: 512, numHeads: 12, selfAttention: true); // throws\n\n// after\nnew MultiHeadAttention(embeddingDim: 768, keyDim: 768, valueDim: 768, numHeads: 12, selfAttention: true); // OK","handlingStrategy":"validation","validationCode":"if (selfAttention && !(embeddingDim == keyDim && keyDim == valueDim))\n    throw new ArgumentException(\"Self-attention requires embeddingDim == keyDim == valueDim.\");","typeGuard":null,"tryCatchPattern":"try { var attn = new MultiHeadAttention(...); }\ncatch (ArgumentException ex) { log.LogError(ex, \"Self-attention dim mismatch\"); throw new ModelConfigException(...); }","preventionTips":["Use the same hiddenSize constant for embeddingDim/keyDim/valueDim in self-attention.","Only set selfAttention:true for encoder self-attention layers; use cross-attention for encoder-decoder.","Double-check constructor parameter order when wiring configs."],"tags":["csharp","attention","dimensions","self-attention"],"backgroundTag":"invalid-argument-value","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}