{"record":{"id":"1646c815dc087a17","repo":"dotnet/machinelearning","slug":"sliding-window-is-not-supported-for-non-causal-mas","errorCode":null,"errorMessage":"Sliding window is not supported for non-causal masks","messagePattern":"Sliding window is not supported for non-causal masks","errorType":"exception","errorClass":"NotImplementedException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.GenAI.Core/Utility/AttentionMaskConverter.cs","lineNumber":58,"sourceCode":"    {\n        long[] inputShape = [attentionMask2d.shape[0], queryLength];\n\n        // create causal mask\n        // [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]\n        Tensor? causal4dMask = null;\n        if ((inputShape[^1] > 1 || this._slidingWindow is not null) && this._isCausal)\n        {\n            if (keyValueLength is null)\n            {\n                throw new ArgumentException(\"key_value_length should be provided when attention_mask is causal\");\n            }\n\n            var pastKeyValuesLength = keyValueLength.Value - queryLength;\n            causal4dMask = MakeCausalMask(inputShape, dType, attentionMask2d.device, pastKeyValuesLength, this._slidingWindow);\n        }\n        else if (this._slidingWindow is not null)\n        {\n            throw new NotImplementedException(\"Sliding window is not supported for non-causal masks\");\n        }\n\n        var expandedAttnMask = ExpandMask(attentionMask2d, dType, queryLength).to(attentionMask2d.device);\n        if (causal4dMask is not null)\n        {\n            var min = torch.finfo(dType).min;\n            expandedAttnMask = causal4dMask.masked_fill(expandedAttnMask.to(ScalarType.Bool), min);\n        }\n\n        return expandedAttnMask;\n    }\n\n    public Tensor? ToCausal4D(\n        int batchSize,\n        int queryLength,\n        int keyValueLength,\n        ScalarType dType,\n        Device device)","sourceCodeStart":40,"sourceCodeEnd":76,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.GenAI.Core/Utility/AttentionMaskConverter.cs#L40-L76","documentation":"AttentionMaskConverter.To4D throws NotImplementedException when a sliding window is configured (_slidingWindow != null) but the mask is not causal. Sliding-window masking is only implemented for causal masks (as in Mistral-style attention), so the non-causal branch explicitly rejects it.","triggerScenarios":"Constructing an AttentionMaskConverter with a sliding window and _isCausal = false, then calling To4D (via Create4DCausalAttentionMask) with seq_len > 1.","commonSituations":"Porting attention code from models like Mistral/Longformer to a non-causal (encoder-style or bidirectional) context; accidentally setting isCausal: false while keeping the sliding-window parameter from a sliding-window model config.","solutions":["Set isCausal: true when using a sliding window in decoder attention.","Remove the sliding-window parameter if your attention is non-causal.","Implement or upstream a MakeSlidingMask path for non-causal masks if you truly need bidirectional sliding-window attention.","Avoid the sliding-window converter and construct the 4D mask manually for encoder use."],"exampleFix":"// before\nvar converter = new AttentionMaskConverter(slidingWindow: 4096, isCausal: false);\n// after\nvar converter = new AttentionMaskConverter(slidingWindow: 4096, isCausal: true);","handlingStrategy":"validation","validationCode":"if (slidingWindow is not null && !isCausal)\n    throw new InvalidOperationException(\"sliding window requires causal attention in this library\");","typeGuard":"bool SlidingWindowOk(int? slidingWindow, bool isCausal) => slidingWindow is null || isCausal;","tryCatchPattern":"try { var mask = converter.To4D(attn2d, seqLen, dType, kvLen); }\ncatch (NotImplementedException ex) when (ex.Message.Contains(\"Sliding window\")) { logger.LogError(\"Disable sliding window or enable causal masking\"); throw; }","preventionTips":["Only use sliding window with causal decoder attention (e.g. Mistral-style).","Read the model config's sliding_window together with is_causal.","For bidirectional attention, drop the sliding-window parameter.","Extend the converter yourself if you need non-causal sliding masks."],"tags":["attention","mask","unsupported"],"backgroundTag":"unsupported-operation","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}