dotnet/machinelearning · error · ArgumentException

num_key_value_heads must be specified

Error message

num_key_value_heads must be specified

What it means

Phi2Attention computes grouped-query attention and needs NumKeyValueHeads from the Phi2 config. If config.NumKeyValueHeads is null (not set when loading a checkpoint/config), the constructor throws ArgumentException('num_key_value_heads must be specified').

Source

Thrown at src/Microsoft.ML.GenAI.Phi/Module/Phi2Attention.cs:64

    private readonly LayerNorm? k_layernorm;

    private readonly Phi2RotaryEmbedding phiRotaryEmbedding;

    // cache_k, cache_v
    private Tensor cache_k;
    private Tensor cache_v;
#pragma warning restore MSML_PrivateFieldName // Private field name not in: _camelCase format

    public Phi2Attention(Phi2Config config, int? layerIdx = null, int maxBatch = 2, int maxLength = 1024)
        : base(nameof(Phi2Attention))
    {
        this._layerIdx = layerIdx;
        this._config = config;
        this._attentionDropout = config.AttentionDropout;
        this._hiddenSize = config.HiddenSize;
        this._numAttentionHeads = config.NumAttentionHeads;
        this._headDim = this._hiddenSize / this._numAttentionHeads;
        this._numKeyValueHeads = config.NumKeyValueHeads ?? throw new ArgumentException("num_key_value_heads must be specified");
        this._numKeyValueGroups = this._numAttentionHeads / this._numKeyValueHeads;
        this._maxPositionEmbeddings = config.MaxPositionEmbeddings;
        this._ropeTheta = config.RopeTheta;
        this._partialRotaryFactor = config.PartialRotaryFactor;

        Contract.Assert(this._hiddenSize % (this._headDim * this._numAttentionHeads) == 0, "hidden_size must be divisible by num_attention_heads");
        this.q_proj = new GenAILinear(this._hiddenSize, this._numAttentionHeads * this._headDim, hasBias: true, dtype: config.Dtype);
        this.k_proj = new GenAILinear(this._hiddenSize, this._numKeyValueHeads * this._headDim, hasBias: true, dtype: config.Dtype);
        this.v_proj = new GenAILinear(this._hiddenSize, this._numKeyValueHeads * this._headDim, hasBias: true, dtype: config.Dtype);
        this.dense = new GenAILinear(this._numAttentionHeads * this._headDim, this._hiddenSize, hasBias: true, dtype: config.Dtype);

        this._qkLayernorm = config.QkLayernorm;
        if (this._qkLayernorm)
        {
            this.q_layernorm = nn.LayerNorm(this._hiddenSize / this._numAttentionHeads, eps: config.LayerNormEps, elementwise_affine: true, dtype: config.Dtype);
            this.k_layernorm = nn.LayerNorm(this._hiddenSize / this._numAttentionHeads, eps: config.LayerNormEps, elementwise_affine: true, dtype: config.Dtype);
        }

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Set NumKeyValueHeads in the config (for Phi-2, equal to NumAttentionHeads)
2. Patch the model's config.json to add "num_key_value_heads" before loading
3. Default it explicitly when building config programmatically: NumKeyValueHeads = NumAttentionHeads

Example fix

// before
var config = new Phi2Config { HiddenSize = 2560, NumAttentionHeads = 32 };
// after
var config = new Phi2Config { HiddenSize = 2560, NumAttentionHeads = 32, NumKeyValueHeads = 32 };

if (config.NumKeyValueHeads is null) config.NumKeyValueHeads = config.NumAttentionHeads;

static bool IsPhi2ConfigComplete(Phi2Config c) => c.NumKeyValueHeads.HasValue && c.NumAttentionHeads > 0;

try { model = new Phi2Model(config); }
catch (ArgumentException ex) when (ex.Message.Contains("num_key_value_heads")) { config.NumKeyValueHeads = config.NumAttentionHeads; model = new Phi2Model(config); }

When it happens

Trigger: Constructing Phi2Attention (e.g. via Phi2Model or CreateAttentionFromConfig) with a config whose num_key_value_heads field is absent — common with original Phi-2 checkpoints that predate GQA fields.

Common situations: Loading a HuggingFace Phi-2 config.json lacking num_key_value_heads; hand-building a Phi2Config without setting NumKeyValueHeads; newer library version requiring GQA where older configs didn't include it.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• inputEmbeddings is not supported
• num_key_value_heads must be specified
• No Onnx Session Options
• Activation function {name} not supported.
• {nameof(ChannelMessageKind)}.{e.Kind} is not yet implemented

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/5d26cbba004499ae. Report an issue: GitHub.