{"record":{"id":"10c090978e3cb741","repo":"dotnet/machinelearning","slug":"dimension-must-be-divisible-by-2","errorCode":null,"errorMessage":"Dimension must be divisible by 2","messagePattern":"Dimension must be divisible by 2","errorType":"validation","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.GenAI.Core/Utils.cs","lineNumber":54,"sourceCode":"        // Console.WriteLine(rotated_complex.mean().ToSingle());\n\n        // Convert the complex number back to the real number\n        // (B, Seq_Len, H, Head_Dim/2) -> (B, Seq_Len, H, Head_Dim/2, 2)\n        var rotated = rotatedComplex.view_as_real();\n\n        // (B, Seq_Len, H, Head_Dim/2, 2) -> (B, Seq_Len, H, Head_Dim)\n        var rotatedReshaped = rotated.reshape(rotated.shape[0], rotated.shape[1], rotated.shape[2], -1);\n\n        return rotatedReshaped.type_as(input);\n    }\n\n    public static Tensor PrecomputeThetaPosFrequencies(int headDim, int seqLen, string device, float theta = 10000.0f)\n    {\n        // As written in the paragraph 3.2.2 of the paper\n        // >> In order to generalize our results in 2D to any xi ∈ Rd where **d is even**, [...]\n        if (headDim % 2 != 0)\n        {\n            throw new ArgumentException(\"Dimension must be divisible by 2\", nameof(headDim));\n        }\n\n        // Build the theta parameter\n        // According to the formula theta_i = 10000^(-2(i-1)/dim) for i = [1, 2, ... dim/2]\n        // Shape: (Head_Dim / 2)\n        var thetaNumerator = torch.arange(0, headDim, 2).to(torch.float32).to(device);\n        // Shape: (Head_Dim / 2)\n        var thetaInput = torch.pow(theta, -1.0f * (thetaNumerator / headDim)).to(device); // (Dim / 2)\n        // Construct the positions (the \"m\" parameter)\n        // Shape: (Seq_Len)\n        var m = torch.arange(seqLen, device: device);\n        // Multiply each theta by each position using the outer product.\n        // Shape: (Seq_Len) outer_product* (Head_Dim / 2) -> (Seq_Len, Head_Dim / 2)\n        var thetaPositionFrequencies = torch.outer(m, thetaInput).to(torch.float32).to(device);\n\n        // We can compute complex numbers in the polar form c = R * exp(m * theta), where R = 1 as follows:\n        // (Seq_Len, Head_Dim / 2) -> (Seq_Len, Head_Dim / 2)\n        var freqsComplex = torch.polar(torch.ones_like(thetaPositionFrequencies), thetaPositionFrequencies);","sourceCodeStart":36,"sourceCodeEnd":72,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.GenAI.Core/Utils.cs#L36-L72","documentation":"Utils.PrecomputeThetaPosFrequencies throws ArgumentException with parameter name headDim when the head dimension is odd. RoPE (Rotary Position Embedding) frequency precomputation requires an even head dimension because frequencies are computed in pairs (theta_i = 10000^(-2(i-1)/dim)) over headDim/2 entries, per paragraph 2.2.2/3.2.2 of the RoFormer paper.","triggerScenarios":"Calling PrecomputeThetaPosFrequencies with an odd headDim, typically because the model's hidden size divided by number of attention heads is odd (e.g. head_dim = 63).","commonSituations":"Custom/quantized model configs where num_heads doesn't evenly divide hidden_size; hand-edited config.json values; experimental architectures with odd head dims.","solutions":["Fix the model config so hidden_size / num_attention_heads is even.","Verify headDim used in your RoPE setup matches the checkpoint's config (head_dim field).","If the architecture genuinely has an odd head dimension, RoPE cannot be applied as-is — pad or choose a different positional encoding.","Double-check you are passing headDim and not hiddenSize to the method."],"exampleFix":"// before\nvar freqs = Utils.PrecomputeThetaPosFrequencies(headDim: hiddenSize / numHeads, seqLen, device); // = 63, odd\n// after\nint headDim = hiddenSize / numHeads; // choose numHeads so this is even, e.g. hiddenSize=4032, numHeads=64 -> 63 invalid; numHeads=72 -> 56 valid\nvar freqs = Utils.PrecomputeThetaPosFrequencies(headDim, seqLen, device);","handlingStrategy":"validation","validationCode":"int headDim = hiddenSize / numHeads;\nif (headDim % 2 != 0) throw new InvalidOperationException($\"headDim {headDim} must be even for RoPE\");","typeGuard":"bool EvenHeadDim(int hiddenSize, int numHeads) => numHeads > 0 && (hiddenSize / numHeads) % 2 == 0;","tryCatchPattern":"try { var freqs = Utils.PrecomputeThetaPosFrequencies(headDim, seqLen, device); }\ncatch (ArgumentException ex) when (ex.ParamName == nameof(headDim)) { logger.LogError(\"headDim must be even; check hidden_size/num_heads in config\"); throw; }","preventionTips":["Validate hidden_size is divisible by num_attention_heads when editing configs.","Pass headDim (per-head), never full hidden size, to PrecomputeThetaPosFrequencies.","Check the checkpoint's head_dim field against your computed value.","Prefer official configs over hand-edited ones."],"tags":["rope","position-embedding","argument"],"backgroundTag":"invalid-argument-value","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}