huggingface/candle · error

compile with '--features flash-attn'

Error message

compile with '--features flash-attn'

What it means

Same pattern as other candle models: gemma3.rs contains a `flash_attn` helper whose `#[cfg(not(feature = "flash-attn"))]` fallback stubs the call with `unimplemented!`. If the model config requests flash attention but the crate lacks the feature, the stub panics at runtime.

Source

Thrown at candle-transformers/src/models/gemma3.rs:305

            KvCache::Rotating(c) => c.reset(),
        }
    }
}

#[cfg(feature = "flash-attn")]
fn flash_attn(
    q: &Tensor,
    k: &Tensor,
    v: &Tensor,
    softmax_scale: f32,
    causal: bool,
) -> Result<Tensor> {
    candle_flash_attn::flash_attn(q, k, v, softmax_scale, causal)
}

#[cfg(not(feature = "flash-attn"))]
fn flash_attn(_: &Tensor, _: &Tensor, _: &Tensor, _: f32, _: bool) -> Result<Tensor> {
    unimplemented!("compile with '--features flash-attn'")
}

#[derive(Debug, Clone)]
struct DecoderLayer {
    self_attn: Attention,
    mlp: MLP,
    input_layernorm: RmsNorm,
    pre_feedforward_layernorm: RmsNorm,
    post_feedforward_layernorm: RmsNorm,
    post_attention_layernorm: RmsNorm,
    sliding_window: Option<usize>,
}

impl DecoderLayer {
    fn new(
        use_flash_attn: bool,
        cfg: &Config,
        vb: VarBuilder,

View on GitHub (pinned to d5fee525bf)

Solutions

1. Rebuild with `--features candle-transformers/flash-attn` on a CUDA-capable machine.
2. Force `config.use_flash_attn = false` before constructing the model so standard SDPA attention is used.
3. Verify the target platform supports candle-flash-attn (CUDA only); otherwise stay on the default path.

Example fix

// before
cargo run --release -- --model gemma3
// after
cargo run --release --features candle-transformers/flash-attn -- --model gemma3

let mut config = gemma3::Config::from_json(json)?;
if config.use_flash_attn && !cfg!(feature = "flash-attn") {
    eprintln!("flash-attn feature not enabled; falling back to standard attention");
    config.use_flash_attn = false;
}

fn can_use_flash_attn(cfg: &gemma3::Config) -> bool {
    !cfg.use_flash_attn || cfg!(feature = "flash-attn")
}

std::panic::catch_unwind(|| model.forward(&input)).map_err(|_| anyhow::anyhow!("flash_attn unimplemented; rebuild with --features candle-transformers/flash-attn"))

When it happens

Trigger: Instantiating/running the Gemma3 text model with `use_flash_attn: true` in Config while compiled without the `flash-attn` cargo feature.

Common situations: Checkpoint config.json enables flash attention; default `cargo run` without features; attempting flash attention on a non-CUDA host where the feature cannot be enabled.

Related errors

• compile with '--features flash-attn'
• compile with '--features flash-attn'
• compile with '--features flash-attn'
• compile with '--features flash-attn'
• compile with '--features flash-attn'

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/dabb5ae75eb73777. Report an issue: GitHub.