dotnet/machinelearning · error · ArgumentException

key_value_length should be provided when attention_mask is…

Error message

key_value_length should be provided when attention_mask is causal

What it means

AttentionMaskConverter.To4D throws ArgumentException when building a 4D causal attention mask (query length > 1 or a sliding window is set and _isCausal is true) without a keyValueLength. The past key-value length (keyValueLength - queryLength) is required to offset the causal mask during cached/decoded generation.

Solutions

1. Pass a valid pastKeyValuesLength (typically past cache sequence length) when seq_len > 1 in generation.
2. Only omit keyValueLength for prefill steps where queryLength equals the full key/value length.
3. Compute keyValueLength as pastKvLength + queryLength before calling Create4DCausalAttentionMask.
4. If decoding step-by-step (queryLength == 1) without sliding window, ensure _isCausal handling matches the HF convention that doesn't need the mask.

Example fix

// before
var mask = AttentionMaskConverter.Create4DCausalAttentionMask(attentionMask, inputShape, dType, pastKeyValuesLength: null, device, isCausal: true);
// after
var mask = AttentionMaskConverter.Create4DCausalAttentionMask(attentionMask, inputShape, dType, pastKeyValuesLength: pastKvLen + inputShape[1], device, isCausal: true);

// before calling Create4DCausalAttentionMask in a KV-cached decode path
if ((seqLen > 1 || slidingWindow is not null) && pastKeyValuesLength is null)
    throw new InvalidOperationException("pass pastKeyValuesLength for causal masking with seqLen>1");

bool HasKvLength(long? kvLen, int seqLen, int? slidingWindow) => seqLen == 1 || kvLen.HasValue || slidingWindow is null;

try { var mask = AttentionMaskConverter.Create4DCausalAttentionMask(attn2d, inputShape, dType, pastKeyValuesLength, device, isCausal: true); }
catch (ArgumentException ex) when (ex.Message.Contains("key_value_length")) { logger.LogError("Supply pastKeyValuesLength during KV-cached decoding"); throw; }

When it happens

Trigger: Calling Create4DCausalAttentionMask/To4D during incremental decoding with a causal mask and seq_len > 1 (or sliding window) while passing null pastKeyValuesLength.

Common situations: Custom attention-code paths written for the prefill-only case (keyValueLength derivable) reused for KV-cached decoding; calling the converter directly instead of through the model's attention layer which always knows the cache length.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• Attention mask should be 2D
• Sliding window is not supported for non-causal masks
• This is not a causal mask
• Either input_ids or inputs_embeds must be set
• EmbeddingDim must be divisible by NumHeads

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/6ac2ddd5fd664fe8. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.GenAI.Core/Utility/AttentionMaskConverter.cs:50

    /// <param name="dType"></param>
    /// <param name="keyValueLength"></param>
    /// <returns></returns>
    public Tensor To4D(
        Tensor attentionMask2d,
        int queryLength,
        ScalarType dType,
        int? keyValueLength = null)
    {
        long[] inputShape = [attentionMask2d.shape[0], queryLength];

        // create causal mask
        // [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]
        Tensor? causal4dMask = null;
        if ((inputShape[^1] > 1 || this._slidingWindow is not null) && this._isCausal)
        {
            if (keyValueLength is null)
            {
                throw new ArgumentException("key_value_length should be provided when attention_mask is causal");
            }

            var pastKeyValuesLength = keyValueLength.Value - queryLength;
            causal4dMask = MakeCausalMask(inputShape, dType, attentionMask2d.device, pastKeyValuesLength, this._slidingWindow);
        }
        else if (this._slidingWindow is not null)
        {
            throw new NotImplementedException("Sliding window is not supported for non-causal masks");
        }

        var expandedAttnMask = ExpandMask(attentionMask2d, dType, queryLength).to(attentionMask2d.device);
        if (causal4dMask is not null)
        {
            var min = torch.finfo(dType).min;
            expandedAttnMask = causal4dMask.masked_fill(expandedAttnMask.to(ScalarType.Bool), min);
        }

        return expandedAttnMask;

View on GitHub (pinned to 7b76e69cf9)