{"record":{"id":"6ac2ddd5fd664fe8","repo":"dotnet/machinelearning","slug":"key-value-length-should-be-provided-when-attention","errorCode":null,"errorMessage":"key_value_length should be provided when attention_mask is causal","messagePattern":"key_value_length should be provided when attention_mask is causal","errorType":"validation","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.GenAI.Core/Utility/AttentionMaskConverter.cs","lineNumber":50,"sourceCode":"    /// <param name=\"dType\"></param>\n    /// <param name=\"keyValueLength\"></param>\n    /// <returns></returns>\n    public Tensor To4D(\n        Tensor attentionMask2d,\n        int queryLength,\n        ScalarType dType,\n        int? keyValueLength = null)\n    {\n        long[] inputShape = [attentionMask2d.shape[0], queryLength];\n\n        // create causal mask\n        // [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]\n        Tensor? causal4dMask = null;\n        if ((inputShape[^1] > 1 || this._slidingWindow is not null) && this._isCausal)\n        {\n            if (keyValueLength is null)\n            {\n                throw new ArgumentException(\"key_value_length should be provided when attention_mask is causal\");\n            }\n\n            var pastKeyValuesLength = keyValueLength.Value - queryLength;\n            causal4dMask = MakeCausalMask(inputShape, dType, attentionMask2d.device, pastKeyValuesLength, this._slidingWindow);\n        }\n        else if (this._slidingWindow is not null)\n        {\n            throw new NotImplementedException(\"Sliding window is not supported for non-causal masks\");\n        }\n\n        var expandedAttnMask = ExpandMask(attentionMask2d, dType, queryLength).to(attentionMask2d.device);\n        if (causal4dMask is not null)\n        {\n            var min = torch.finfo(dType).min;\n            expandedAttnMask = causal4dMask.masked_fill(expandedAttnMask.to(ScalarType.Bool), min);\n        }\n\n        return expandedAttnMask;","sourceCodeStart":32,"sourceCodeEnd":68,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.GenAI.Core/Utility/AttentionMaskConverter.cs#L32-L68","documentation":"AttentionMaskConverter.To4D throws ArgumentException when building a 4D causal attention mask (query length > 1 or a sliding window is set and _isCausal is true) without a keyValueLength. The past key-value length (keyValueLength - queryLength) is required to offset the causal mask during cached/decoded generation.","triggerScenarios":"Calling Create4DCausalAttentionMask/To4D during incremental decoding with a causal mask and seq_len > 1 (or sliding window) while passing null pastKeyValuesLength.","commonSituations":"Custom attention-code paths written for the prefill-only case (keyValueLength derivable) reused for KV-cached decoding; calling the converter directly instead of through the model's attention layer which always knows the cache length.","solutions":["Pass a valid pastKeyValuesLength (typically past cache sequence length) when seq_len > 1 in generation.","Only omit keyValueLength for prefill steps where queryLength equals the full key/value length.","Compute keyValueLength as pastKvLength + queryLength before calling Create4DCausalAttentionMask.","If decoding step-by-step (queryLength == 1) without sliding window, ensure _isCausal handling matches the HF convention that doesn't need the mask."],"exampleFix":"// before\nvar mask = AttentionMaskConverter.Create4DCausalAttentionMask(attentionMask, inputShape, dType, pastKeyValuesLength: null, device, isCausal: true);\n// after\nvar mask = AttentionMaskConverter.Create4DCausalAttentionMask(attentionMask, inputShape, dType, pastKeyValuesLength: pastKvLen + inputShape[1], device, isCausal: true);","handlingStrategy":"validation","validationCode":"// before calling Create4DCausalAttentionMask in a KV-cached decode path\nif ((seqLen > 1 || slidingWindow is not null) && pastKeyValuesLength is null)\n    throw new InvalidOperationException(\"pass pastKeyValuesLength for causal masking with seqLen>1\");","typeGuard":"bool HasKvLength(long? kvLen, int seqLen, int? slidingWindow) => seqLen == 1 || kvLen.HasValue || slidingWindow is null;","tryCatchPattern":"try { var mask = AttentionMaskConverter.Create4DCausalAttentionMask(attn2d, inputShape, dType, pastKeyValuesLength, device, isCausal: true); }\ncatch (ArgumentException ex) when (ex.Message.Contains(\"key_value_length\")) { logger.LogError(\"Supply pastKeyValuesLength during KV-cached decoding\"); throw; }","preventionTips":["Always track past KV cache length in generation loops and pass it through.","Omit the mask only for true single-token decode steps.","Keep prefill and decode mask-building paths separate.","Match HF Transformers conventions when porting attention code."],"tags":["attention","mask","tensor"],"backgroundTag":"missing-required-argument","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}