dotnet/machinelearning · error · ArgumentException

Self-attention requires query, key and value to be of the…

Error message

Self-attention requires query, key and value to be of the same size

What it means

In self-attention the same tensor serves as query, key, and value, so their embedding dimensions must be equal; MultiHeadAttention throws ArgumentException when selfAttention=true but the supplied q/k/v dimensions differ (_qkvSameDim is false). This prevents constructing a module whose weights could never be shared/applied consistently.

Solutions

1. For self-attention, pass embeddingDim, keyDim, and valueDim as the same value (commonly all equal to hiddenSize).
2. Set selfAttention: false if you actually need cross-attention with differing q/k/v dims (as in encoder-decoder attention).
3. Validate dims before construction: `if (selfAttention && !(embeddingDim == keyDim && keyDim == valueDim)) throw ...`.
4. Wrap the constructor call in try-catch on ArgumentException to surface a configuration error.

Example fix

// before
new MultiHeadAttention(embeddingDim: 768, keyDim: 512, valueDim: 512, numHeads: 12, selfAttention: true); // throws

// after
new MultiHeadAttention(embeddingDim: 768, keyDim: 768, valueDim: 768, numHeads: 12, selfAttention: true); // OK

if (selfAttention && !(embeddingDim == keyDim && keyDim == valueDim))
    throw new ArgumentException("Self-attention requires embeddingDim == keyDim == valueDim.");

try { var attn = new MultiHeadAttention(...); }
catch (ArgumentException ex) { log.LogError(ex, "Self-attention dim mismatch"); throw new ModelConfigException(...); }

When it happens

Trigger: Calling the MultiHeadAttention constructor with selfAttention: true while passing different embeddingDim, keyDim, and valueDim values (e.g., embeddingDim=768, keyDim=512).

Common situations: Reusing a cross-attention configuration (different q/k/v sizes) in a self-attention slot, or misreading constructor parameter order and passing keyDim/valueDim where embeddingDim was expected.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• EmbeddingDim must be divisible by NumHeads
• NumAttentionHeads must be a factor of hiddenSize, got
• Invalid height value.
• Invalid width value.
• A PrimitiveDataViewType cannot have a disposable RawType

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/a0274ac5e521dbda. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.TorchSharp/NasBert/Modules/MultiHeadAttention.cs:84

            _embeddingDim = embeddingDim;
            _kDim = kDim ?? embeddingDim;
            _vDim = vDim ?? embeddingDim;
            _qkvSameDim = (_kDim == _embeddingDim) && (_vDim == _embeddingDim);

            _numHeads = numHeads;
            _dropout = dropout;
            _headDim = _embeddingDim / _numHeads;
            _scaling = Math.Pow(_headDim, -0.5);
            if (_headDim * _numHeads != _embeddingDim)
            {
                throw new ArgumentException("EmbeddingDim must be divisible by NumHeads");
            }

            _selfAttention = selfAttention;
            _encoderDecoderAttention = encoderDecoderAttention;
            if (_selfAttention && !_qkvSameDim)
            {
                throw new ArgumentException("Self-attention requires query, key and value to be of the same size");
            }

            _addBiasProj = bias;
            _addBiasKv = addBiasKv;
            _addZeroAttention = addZeroAttention;

            QProjection = torch.nn.Linear(_embeddingDim, _embeddingDim, _addBiasProj);
            KProjection = torch.nn.Linear(_kDim, _embeddingDim, _addBiasProj);
            VProjection = torch.nn.Linear(_vDim, _embeddingDim, _addBiasProj);

            if (_addBiasKv)
            {
                KBias = torch.zeros(1, 1, _embeddingDim).AsParameter();
                VBias = torch.zeros(1, 1, _embeddingDim).AsParameter();
            }

            OutProjLinear = torch.nn.Linear(_embeddingDim, _embeddingDim, _addBiasProj);
            DropoutLayer = torch.nn.Dropout(_dropout);

View on GitHub (pinned to 7b76e69cf9)