dotnet/machinelearning · error · ArgumentException

Dimension must be divisible by 2

Error message

Dimension must be divisible by 2

What it means

Utils.PrecomputeThetaPosFrequencies throws ArgumentException with parameter name headDim when the head dimension is odd. RoPE (Rotary Position Embedding) frequency precomputation requires an even head dimension because frequencies are computed in pairs (theta_i = 10000^(-2(i-1)/dim)) over headDim/2 entries, per paragraph 2.2.2/3.2.2 of the RoFormer paper.

Source

Thrown at src/Microsoft.ML.GenAI.Core/Utils.cs:54

        // Console.WriteLine(rotated_complex.mean().ToSingle());

        // Convert the complex number back to the real number
        // (B, Seq_Len, H, Head_Dim/2) -> (B, Seq_Len, H, Head_Dim/2, 2)
        var rotated = rotatedComplex.view_as_real();

        // (B, Seq_Len, H, Head_Dim/2, 2) -> (B, Seq_Len, H, Head_Dim)
        var rotatedReshaped = rotated.reshape(rotated.shape[0], rotated.shape[1], rotated.shape[2], -1);

        return rotatedReshaped.type_as(input);
    }

    public static Tensor PrecomputeThetaPosFrequencies(int headDim, int seqLen, string device, float theta = 10000.0f)
    {
        // As written in the paragraph 3.2.2 of the paper
        // >> In order to generalize our results in 2D to any xi ∈ Rd where **d is even**, [...]
        if (headDim % 2 != 0)
        {
            throw new ArgumentException("Dimension must be divisible by 2", nameof(headDim));
        }

        // Build the theta parameter
        // According to the formula theta_i = 10000^(-2(i-1)/dim) for i = [1, 2, ... dim/2]
        // Shape: (Head_Dim / 2)
        var thetaNumerator = torch.arange(0, headDim, 2).to(torch.float32).to(device);
        // Shape: (Head_Dim / 2)
        var thetaInput = torch.pow(theta, -1.0f * (thetaNumerator / headDim)).to(device); // (Dim / 2)
        // Construct the positions (the "m" parameter)
        // Shape: (Seq_Len)
        var m = torch.arange(seqLen, device: device);
        // Multiply each theta by each position using the outer product.
        // Shape: (Seq_Len) outer_product* (Head_Dim / 2) -> (Seq_Len, Head_Dim / 2)
        var thetaPositionFrequencies = torch.outer(m, thetaInput).to(torch.float32).to(device);

        // We can compute complex numbers in the polar form c = R * exp(m * theta), where R = 1 as follows:
        // (Seq_Len, Head_Dim / 2) -> (Seq_Len, Head_Dim / 2)
        var freqsComplex = torch.polar(torch.ones_like(thetaPositionFrequencies), thetaPositionFrequencies);

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Fix the model config so hidden_size / num_attention_heads is even.
2. Verify headDim used in your RoPE setup matches the checkpoint's config (head_dim field).
3. If the architecture genuinely has an odd head dimension, RoPE cannot be applied as-is — pad or choose a different positional encoding.
4. Double-check you are passing headDim and not hiddenSize to the method.

Example fix

// before
var freqs = Utils.PrecomputeThetaPosFrequencies(headDim: hiddenSize / numHeads, seqLen, device); // = 63, odd
// after
int headDim = hiddenSize / numHeads; // choose numHeads so this is even, e.g. hiddenSize=4032, numHeads=64 -> 63 invalid; numHeads=72 -> 56 valid
var freqs = Utils.PrecomputeThetaPosFrequencies(headDim, seqLen, device);

int headDim = hiddenSize / numHeads;
if (headDim % 2 != 0) throw new InvalidOperationException($"headDim {headDim} must be even for RoPE");

bool EvenHeadDim(int hiddenSize, int numHeads) => numHeads > 0 && (hiddenSize / numHeads) % 2 == 0;

try { var freqs = Utils.PrecomputeThetaPosFrequencies(headDim, seqLen, device); }
catch (ArgumentException ex) when (ex.ParamName == nameof(headDim)) { logger.LogError("headDim must be even; check hidden_size/num_heads in config"); throw; }

When it happens

Trigger: Calling PrecomputeThetaPosFrequencies with an odd headDim, typically because the model's hidden size divided by number of attention heads is odd (e.g. head_dim = 63).

Common situations: Custom/quantized model configs where num_heads doesn't evenly divide hidden_size; hand-edited config.json values; experimental architectures with odd head dims.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• Exception of type 'System.ArgumentException' was thrown.
• Expected either {0} or {1} to be provided
• Expected a seekable stream
• Decimal separator cannot match the column separator
• Array lengths are mistmached

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/10c090978e3cb741. Report an issue: GitHub.