dotnet/machinelearning · error · ArgumentException

NumAttentionHeads must be a factor of hiddenSize, got

Error message

NumAttentionHeads must be a factor of hiddenSize, got {numAttentionHeads} and {hiddenSize}.

What it means

AttentionSelf divides hiddenSize by numAttentionHeads to get each head's size and throws ArgumentException when numAttentionHeads * AttentionHeadSize != hiddenSize, i.e., when the head count is not an exact factor of hiddenSize. Like the MultiHeadAttention check, this enforces dimension consistency at module construction.

Solutions

Pick numAttentionHeads that divides hiddenSize exactly (standard RoBERTa: 768/12, 1024/16).
Validate before construction: `if (hiddenSize % numAttentionHeads != 0) throw ...` in your options-validation code.
Derive AttentionHeadSize after validating rather than relying on integer division.
Wrap construction in try-catch on ArgumentException and report the two offending values to the caller.

Example fix

// before
new AttentionSelf(hiddenSize: 768, numAttentionHeads: 10); // 10 * 76 != 768, throws

// after
new AttentionSelf(hiddenSize: 768, numAttentionHeads: 12); // 12 * 64 == 768

Defensive patterns

Strategy: validation

Validate before calling

if (hiddenSize % numAttentionHeads != 0)
    throw new ArgumentException($"hiddenSize ({hiddenSize}) must be divisible by numAttentionHeads ({numAttentionHeads}).");

Try / catch

try { var layer = new AttentionSelf(...); }
catch (ArgumentException ex) { log.LogError(ex, "Invalid Roberta attention config"); throw new ModelConfigException(...); }

Prevention

Keep architecture options consistent with the checkpoint's actual dims.
Recompute head counts when shrinking/distilling a model.
Unit-test options objects with representative (hiddenSize, numHeads) pairs.

When it happens

Trigger: Constructing AttentionSelf with numAttentionHeads that does not evenly divide hiddenSize — e.g., hiddenSize=768 with numAttentionHeads=10, or hiddenSize/numHeads using integer division silently truncating before the check catches it.

Common situations: Custom Roberta architecture options where hiddenSize was changed (distilled/smaller models) but numAttentionHeads left at the original value, or config values loaded from an incompatible checkpoint's settings.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/173ecbc0504cdf4d. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.TorchSharp/Roberta/Modules/AttentionSelf.cs:34

        public readonly int NumAttentionHeads;
        public readonly int AttentionHeadSize;

#pragma warning disable MSML_GeneralName // This name should be PascalCased
        public readonly Linear query;
        public readonly Linear key;
        public readonly Linear value;
        public readonly Dropout attention_dropout;

        private bool _disposedValue;

        public AttentionSelf(int numAttentionHeads, long hiddenSize, double layerNormEps, double attentionDropoutRate)
            : base(nameof(AttentionSelf))
        {
            NumAttentionHeads = numAttentionHeads;
            AttentionHeadSize = (int)hiddenSize / numAttentionHeads;
            if (NumAttentionHeads * AttentionHeadSize != hiddenSize)
            {
                throw new ArgumentException($"NumAttentionHeads must be a factor of hiddenSize, got {numAttentionHeads} and {hiddenSize}.");
            }

            query = torch.nn.Linear(hiddenSize, hiddenSize, true);
            key = torch.nn.Linear(hiddenSize, hiddenSize, true);
            value = torch.nn.Linear(hiddenSize, hiddenSize, true);
            attention_dropout = torch.nn.Dropout(attentionDropoutRate);

            RegisterComponents();
        }

        public override torch.Tensor forward(torch.Tensor hiddenStates, torch.Tensor attentionMask)
        {
            using var disposeScope = torch.NewDisposeScope();
            var mixedQueryLayer = query.forward(hiddenStates);
            var mixedKeyLayer = key.forward(hiddenStates);
            var mixedValueLayer = value.forward(hiddenStates);

            var queryLayer = TransposeForScores(mixedQueryLayer);

View on GitHub (pinned to 7b76e69cf9)