dotnet/machinelearning · error · ArgumentException

EmbeddingDim must be divisible by NumHeads

Error message

EmbeddingDim must be divisible by NumHeads

What it means

MultiHeadAttention splits the embedding dimension across attention heads (headDim = embeddingDim / numHeads); if headDim * numHeads != embeddingDim, the heads cannot cover the full embedding dimension and the module throws ArgumentException. This is a constructor-time invariant check that the model dimensions are internally consistent.

Solutions

1. Choose numHeads that divides embeddingDim evenly (e.g., embeddingDim=768 with numHeads=12 gives headDim=64).
2. Validate before construction: `if (embeddingDim % numHeads != 0) throw ...` in your config-loading code.
3. Derive numHeads from embeddingDim programmatically (e.g., fixed headDim target) instead of hard-coding both.
4. Wrap the constructor call in try-catch on ArgumentException to translate it into a config validation error.

Example fix

// before
new MultiHeadAttention(embeddingDim: 100, numHeads: 8); // throws

// after
new MultiHeadAttention(embeddingDim: 768, numHeads: 12); // 768 % 12 == 0

if (embeddingDim % numHeads != 0)
    throw new ArgumentException($"embeddingDim ({embeddingDim}) must be divisible by numHeads ({numHeads}).");

try { var attn = new MultiHeadAttention(...); }
catch (ArgumentException ex) { log.LogError(ex, "Invalid attention dimensions"); throw new ModelConfigException(...); }

When it happens

Trigger: Calling the MultiHeadAttention constructor with embeddingDim that is not an exact multiple of numHeads, e.g. embeddingDim=100, numHeads=8, or a mismatched options object where hiddenSize and numAttentionHeads are configured independently.

Common situations: Hand-editing model architecture options (e.g., shrinking hiddenSize for a smaller model without adjusting head count), porting config JSON from another model whose dims differ, or computing numHeads dynamically with a rounding error.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• NumAttentionHeads must be a factor of hiddenSize, got
• Self-attention requires query, key and value to be of the…
• Invalid height value.
• Invalid width value.
• Stream should be non-null and its stream.CanRead property…

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/6ca03dae2b904f19. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.TorchSharp/NasBert/Modules/MultiHeadAttention.cs:77

            bool bias = true,
            bool addBiasKv = false,
            bool addZeroAttention = false,
            bool selfAttention = false,
            bool encoderDecoderAttention = false)
            : base(nameof(MultiHeadAttention))
        {
            _embeddingDim = embeddingDim;
            _kDim = kDim ?? embeddingDim;
            _vDim = vDim ?? embeddingDim;
            _qkvSameDim = (_kDim == _embeddingDim) && (_vDim == _embeddingDim);

            _numHeads = numHeads;
            _dropout = dropout;
            _headDim = _embeddingDim / _numHeads;
            _scaling = Math.Pow(_headDim, -0.5);
            if (_headDim * _numHeads != _embeddingDim)
            {
                throw new ArgumentException("EmbeddingDim must be divisible by NumHeads");
            }

            _selfAttention = selfAttention;
            _encoderDecoderAttention = encoderDecoderAttention;
            if (_selfAttention && !_qkvSameDim)
            {
                throw new ArgumentException("Self-attention requires query, key and value to be of the same size");
            }

            _addBiasProj = bias;
            _addBiasKv = addBiasKv;
            _addZeroAttention = addZeroAttention;

            QProjection = torch.nn.Linear(_embeddingDim, _embeddingDim, _addBiasProj);
            KProjection = torch.nn.Linear(_kDim, _embeddingDim, _addBiasProj);
            VProjection = torch.nn.Linear(_vDim, _embeddingDim, _addBiasProj);

            if (_addBiasKv)

View on GitHub (pinned to 7b76e69cf9)