What causes "Sliding window is not supported for non-causal masks" in dotnet/machinelearning?

Set isCausal: true when using a sliding window in decoder attention.

dotnet/machinelearning · error · NotImplementedException

Sliding window is not supported for non-causal masks

Error message

Sliding window is not supported for non-causal masks

What it means

AttentionMaskConverter.To4D throws NotImplementedException when a sliding window is configured (_slidingWindow != null) but the mask is not causal. Sliding-window masking is only implemented for causal masks (as in Mistral-style attention), so the non-causal branch explicitly rejects it.

Solutions

Set isCausal: true when using a sliding window in decoder attention.
Remove the sliding-window parameter if your attention is non-causal.
Implement or upstream a MakeSlidingMask path for non-causal masks if you truly need bidirectional sliding-window attention.
Avoid the sliding-window converter and construct the 4D mask manually for encoder use.

Example fix

// before
var converter = new AttentionMaskConverter(slidingWindow: 4096, isCausal: false);
// after
var converter = new AttentionMaskConverter(slidingWindow: 4096, isCausal: true);

Defensive patterns

Strategy: validation

Validate before calling

if (slidingWindow is not null && !isCausal)
    throw new InvalidOperationException("sliding window requires causal attention in this library");

Type guard

bool SlidingWindowOk(int? slidingWindow, bool isCausal) => slidingWindow is null || isCausal;

Try / catch

try { var mask = converter.To4D(attn2d, seqLen, dType, kvLen); }
catch (NotImplementedException ex) when (ex.Message.Contains("Sliding window")) { logger.LogError("Disable sliding window or enable causal masking"); throw; }

Prevention

Only use sliding window with causal decoder attention (e.g. Mistral-style).
Read the model config's sliding_window together with is_causal.
For bidirectional attention, drop the sliding-window parameter.
Extend the converter yourself if you need non-causal sliding masks.

When it happens

Trigger: Constructing an AttentionMaskConverter with a sliding window and _isCausal = false, then calling To4D (via Create4DCausalAttentionMask) with seq_len > 1.

Common situations: Porting attention code from models like Mistral/Longformer to a non-causal (encoder-style or bidirectional) context; accidentally setting isCausal: false while keeping the sliding-window parameter from a sliding-window model config.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/1646c815dc087a17. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.GenAI.Core/Utility/AttentionMaskConverter.cs:58

    {
        long[] inputShape = [attentionMask2d.shape[0], queryLength];

        // create causal mask
        // [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]
        Tensor? causal4dMask = null;
        if ((inputShape[^1] > 1 || this._slidingWindow is not null) && this._isCausal)
        {
            if (keyValueLength is null)
            {
                throw new ArgumentException("key_value_length should be provided when attention_mask is causal");
            }

            var pastKeyValuesLength = keyValueLength.Value - queryLength;
            causal4dMask = MakeCausalMask(inputShape, dType, attentionMask2d.device, pastKeyValuesLength, this._slidingWindow);
        }
        else if (this._slidingWindow is not null)
        {
            throw new NotImplementedException("Sliding window is not supported for non-causal masks");
        }

        var expandedAttnMask = ExpandMask(attentionMask2d, dType, queryLength).to(attentionMask2d.device);
        if (causal4dMask is not null)
        {
            var min = torch.finfo(dType).min;
            expandedAttnMask = causal4dMask.masked_fill(expandedAttnMask.to(ScalarType.Bool), min);
        }

        return expandedAttnMask;
    }

    public Tensor? ToCausal4D(
        int batchSize,
        int queryLength,
        int keyValueLength,
        ScalarType dType,
        Device device)

View on GitHub (pinned to 7b76e69cf9)