dotnet/machinelearning · error · ArgumentException

Dimension must be divisible by 2

Error message

Dimension must be divisible by 2

What it means

PrecomputeThetaPosFrequencies computes RoPE (rotary position embedding) frequency tables for the Phi model. The paper's formula theta_i = 10000^(-2(i-1)/dim) requires an even head dimension so frequencies can be built in headDim/2 pairs, so the method rejects odd headDim with an ArgumentException naming the headDim parameter.

Source

Thrown at src/Microsoft.ML.GenAI.Phi/Utils.cs:25

using System.Linq;
using System.Reflection;
using System.Text;
using System.Threading.Tasks;
using TorchSharp;
using TorchSharp.Modules;
using static TorchSharp.torch;
using static TorchSharp.torch.nn;
namespace Microsoft.ML.GenAI.Phi;

internal static class Utils
{
    public static Tensor PrecomputeThetaPosFrequencies(int headDim, int seqLen, string device, float theta = 10000.0f)
    {
        // As written in the paragraph 3.2.2 of the paper
        // >> In order to generalize our results in 2D to any xi ∈ Rd where **d is even**, [...]
        if (headDim % 2 != 0)
        {
            throw new ArgumentException("Dimension must be divisible by 2", nameof(headDim));
        }

        // Build the theta parameter
        // According to the formula theta_i = 10000^(-2(i-1)/dim) for i = [1, 2, ... dim/2]
        // Shape: (Head_Dim / 2)
        var thetaNumerator = torch.arange(0, headDim, 2).to(torch.float32).to(device);
        // Shape: (Head_Dim / 2)
        var thetaInput = torch.pow(theta, -1.0f * (thetaNumerator / headDim)).to(device); // (Dim / 2)
        // Construct the positions (the "m" parameter)
        // Shape: (Seq_Len)
        var m = torch.arange(seqLen, device: device);
        // Multiply each theta by each position using the outer product.
        // Shape: (Seq_Len) outer_product* (Head_Dim / 2) -> (Seq_Len, Head_Dim / 2)
        var freqs = torch.outer(m, thetaInput).to(torch.float32).to(device);

        // We can compute complex numbers in the polar form c = R * exp(m * theta), where R = 1 as follows:
        // (Seq_Len, Head_Dim / 2) -> (Seq_Len, Head_Dim / 2)
        var freqsComplex = torch.polar(torch.ones_like(freqs), freqs);

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Pass the per-head attention dimension (must be even, e.g. 64, 80, 128), not the total hidden size
2. Verify model config: headDim = hiddenSize / numAttentionHeads and confirm it is even
3. If a custom checkpoint truly has odd headDim, this RoPE implementation cannot be used as-is; pad or use a different implementation

Example fix

// before
Tensor freqs = PrecomputeThetaPosFrequencies(modelConfig.HiddenSize, seqLen, device);
// after
int headDim = modelConfig.HiddenSize / modelConfig.NumAttentionHeads; // e.g. 4096/32 = 128
Tensor freqs = PrecomputeThetaPosFrequencies(headDim, seqLen, device);

if (headDim % 2 != 0)
    throw new ArgumentException($"headDim must be even for RoPE; got {headDim}", nameof(headDim));

bool IsValidHeadDim(int headDim) => headDim > 0 && headDim % 2 == 0;

try
{
    freqs = PrecomputeThetaPosFrequencies(headDim, seqLen, device);
}
catch (ArgumentException ex) when (ex.ParamName == "headDim")
{
    logger.LogError(ex, "Invalid headDim {HeadDim} for RoPE", headDim);
}

When it happens

Trigger: Calling PrecomputeThetaPosFrequencies with a headDim argument that is not divisible by 2, e.g. passing a hidden size, embedding size, or an odd model config value instead of the per-head dimension.

Common situations: Hand-configuring a custom Phi/LLaMA-style model where head_dim was changed or derived incorrectly (e.g. total dim / heads yielding an odd number); porting weights from a nonstandard checkpoint.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• Either input_ids or inputs_embeds must be set
• Directory "{0}" does not exist.
• Unsupported pixel format
• Invalid width value.
• Invalid height value.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/5c71f823461fefc6. Report an issue: GitHub.