What causes "sliding window is not supported" in huggingface/candle?

Use a Qwen3 model/config variant that does not enable sliding window (use_sliding_window: false)

sliding window is not supported

Error message

sliding window is not supported

What it means

Thrown by Qwen3Attention::new when the model config has use_sliding_window set to true. Candle's Qwen3 implementation does not implement sliding-window attention, so it refuses to construct the attention layer rather than silently producing wrong results.

Source

Thrown at candle-transformers/src/models/qwen3.rs:127

    // hyper params
    num_heads: usize,
    num_kv_heads: usize,
    num_kv_groups: usize,
    head_dim: usize,
    hidden_size: usize,
    // utils
    rotary_emb: Arc<Qwen3RotaryEmbedding>,
    kv_cache: ConcatKvCache,
}

impl Qwen3Attention {
    pub(crate) fn new(
        cfg: &Config,
        rotary_emb: Arc<Qwen3RotaryEmbedding>,
        vb: VarBuilder,
    ) -> Result<Self> {
        if cfg.use_sliding_window {
            candle::bail!("sliding window is not supported")
        }

        let head_dim = cfg.head_dim;
        let num_heads = cfg.num_attention_heads;
        let num_kv_heads = cfg.num_key_value_heads;
        let num_kv_groups = num_heads / num_kv_heads;

        let q_proj = linear_b(
            cfg.hidden_size,
            num_heads * head_dim,
            cfg.attention_bias,
            vb.pp("q_proj"),
        )?;
        let k_proj = linear_b(
            cfg.hidden_size,
            num_kv_heads * head_dim,
            cfg.attention_bias,
            vb.pp("k_proj"),

View on GitHub (pinned to d5fee525bf)

Solutions

Use a Qwen3 model/config variant that does not enable sliding window (use_sliding_window: false)
Find or implement sliding-window support in the qwen3 model code, or use a different candle model implementation that supports it
Edit the loaded Config to set use_sliding_window = false only if you understand the accuracy implications for long contexts

Example fix

// before
let mut cfg: Config = serde_json::from_reader(&mut f)?;
cfg.use_sliding_window = true;
let model = qwen3::Model::new(&cfg, vb)?;
// after
let cfg: Config = serde_json::from_reader(&mut f)?;
debug_assert!(!cfg.use_sliding_window, "sliding window unsupported");
let model = qwen3::Model::new(&cfg, vb)?;

Defensive patterns

Strategy: validation

Validate before calling

if cfg.use_sliding_window {
    anyhow::bail!("this Qwen3 config needs sliding-window attention, unsupported here");
}

Try / catch

match qwen3::Model::new(&cfg, vb) {
    Ok(m) => m,
    Err(e) if e.to_string().contains("sliding window") => {
        anyhow::bail!("pick a non sliding-window Qwen3 variant or patch the model impl")
    }
    Err(e) => return Err(e.into()),
}

Prevention

Check cfg.use_sliding_window right after deserializing the config
Choose Qwen3 checkpoints whose configs disable sliding window
Track candle upstream for sliding-window support before adopting those variants

When it happens

Trigger: Instantiating qwen3::Model (or the attention layer directly via new) with a Config where use_sliding_window: true, typically loaded from a Qwen3 config.json of a model variant that uses interleaved sliding-window attention.

Common situations: Loading a Qwen3 long-context variant whose config enables sliding window; hand-editing or copying a config from Qwen2 with sliding window enabled; deserializing a config.json from HuggingFace that includes sliding_window parameters.

Related errors

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/00ba9ea25f6f3e2e. Report an issue: GitHub.