{"record":{"id":"173ecbc0504cdf4d","repo":"dotnet/machinelearning","slug":"numattentionheads-must-be-a-factor-of-hiddensize","errorCode":null,"errorMessage":"NumAttentionHeads must be a factor of hiddenSize, got {numAttentionHeads} and {hiddenSize}.","messagePattern":"NumAttentionHeads must be a factor of hiddenSize, got (.+?) and (.+?)\\.","errorType":"exception","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.TorchSharp/Roberta/Modules/AttentionSelf.cs","lineNumber":34,"sourceCode":"        public readonly int NumAttentionHeads;\n        public readonly int AttentionHeadSize;\n\n#pragma warning disable MSML_GeneralName // This name should be PascalCased\n        public readonly Linear query;\n        public readonly Linear key;\n        public readonly Linear value;\n        public readonly Dropout attention_dropout;\n\n        private bool _disposedValue;\n\n        public AttentionSelf(int numAttentionHeads, long hiddenSize, double layerNormEps, double attentionDropoutRate)\n            : base(nameof(AttentionSelf))\n        {\n            NumAttentionHeads = numAttentionHeads;\n            AttentionHeadSize = (int)hiddenSize / numAttentionHeads;\n            if (NumAttentionHeads * AttentionHeadSize != hiddenSize)\n            {\n                throw new ArgumentException($\"NumAttentionHeads must be a factor of hiddenSize, got {numAttentionHeads} and {hiddenSize}.\");\n            }\n\n            query = torch.nn.Linear(hiddenSize, hiddenSize, true);\n            key = torch.nn.Linear(hiddenSize, hiddenSize, true);\n            value = torch.nn.Linear(hiddenSize, hiddenSize, true);\n            attention_dropout = torch.nn.Dropout(attentionDropoutRate);\n\n            RegisterComponents();\n        }\n\n        public override torch.Tensor forward(torch.Tensor hiddenStates, torch.Tensor attentionMask)\n        {\n            using var disposeScope = torch.NewDisposeScope();\n            var mixedQueryLayer = query.forward(hiddenStates);\n            var mixedKeyLayer = key.forward(hiddenStates);\n            var mixedValueLayer = value.forward(hiddenStates);\n\n            var queryLayer = TransposeForScores(mixedQueryLayer);","sourceCodeStart":16,"sourceCodeEnd":52,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.TorchSharp/Roberta/Modules/AttentionSelf.cs#L16-L52","documentation":"AttentionSelf divides hiddenSize by numAttentionHeads to get each head's size and throws ArgumentException when numAttentionHeads * AttentionHeadSize != hiddenSize, i.e., when the head count is not an exact factor of hiddenSize. Like the MultiHeadAttention check, this enforces dimension consistency at module construction.","triggerScenarios":"Constructing AttentionSelf with numAttentionHeads that does not evenly divide hiddenSize — e.g., hiddenSize=768 with numAttentionHeads=10, or hiddenSize/numHeads using integer division silently truncating before the check catches it.","commonSituations":"Custom Roberta architecture options where hiddenSize was changed (distilled/smaller models) but numAttentionHeads left at the original value, or config values loaded from an incompatible checkpoint's settings.","solutions":["Pick numAttentionHeads that divides hiddenSize exactly (standard RoBERTa: 768/12, 1024/16).","Validate before construction: `if (hiddenSize % numAttentionHeads != 0) throw ...` in your options-validation code.","Derive AttentionHeadSize after validating rather than relying on integer division.","Wrap construction in try-catch on ArgumentException and report the two offending values to the caller."],"exampleFix":"// before\nnew AttentionSelf(hiddenSize: 768, numAttentionHeads: 10); // 10 * 76 != 768, throws\n\n// after\nnew AttentionSelf(hiddenSize: 768, numAttentionHeads: 12); // 12 * 64 == 768","handlingStrategy":"validation","validationCode":"if (hiddenSize % numAttentionHeads != 0)\n    throw new ArgumentException($\"hiddenSize ({hiddenSize}) must be divisible by numAttentionHeads ({numAttentionHeads}).\");","typeGuard":null,"tryCatchPattern":"try { var layer = new AttentionSelf(...); }\ncatch (ArgumentException ex) { log.LogError(ex, \"Invalid Roberta attention config\"); throw new ModelConfigException(...); }","preventionTips":["Keep architecture options consistent with the checkpoint's actual dims.","Recompute head counts when shrinking/distilling a model.","Unit-test options objects with representative (hiddenSize, numHeads) pairs."],"tags":["csharp","attention","dimensions","argument-exception"],"backgroundTag":"invalid-argument-value","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}