huggingface/candle · error

compile with '--features flash-attn'

Error message

compile with '--features flash-attn'

What it means

In candle-transformers, each model defines a `flash_attn` helper. When the crate is compiled without the `flash-attn` cargo feature, a `#[cfg(not(feature = "flash-attn"))]` stub is used that immediately panics via `unimplemented!`. The model code still calls `flash_attn` when flash attention is enabled at runtime, so the panic fires instead of computing attention.

Source

Thrown at candle-transformers/src/models/gemma2.rs:279

    fn clear_kv_cache(&mut self) {
        self.kv_cache = None
    }
}

#[cfg(feature = "flash-attn")]
fn flash_attn(
    q: &Tensor,
    k: &Tensor,
    v: &Tensor,
    softmax_scale: f32,
    causal: bool,
) -> Result<Tensor> {
    candle_flash_attn::flash_attn(q, k, v, softmax_scale, causal)
}

#[cfg(not(feature = "flash-attn"))]
fn flash_attn(_: &Tensor, _: &Tensor, _: &Tensor, _: f32, _: bool) -> Result<Tensor> {
    unimplemented!("compile with '--features flash-attn'")
}

#[derive(Debug, Clone)]
struct DecoderLayer {
    self_attn: Attention,
    mlp: MLP,
    input_layernorm: RmsNorm,
    pre_feedforward_layernorm: RmsNorm,
    post_feedforward_layernorm: RmsNorm,
    post_attention_layernorm: RmsNorm,
}

impl DecoderLayer {
    fn new(
        rotary_emb: Arc<RotaryEmbedding>,
        use_flash_attn: bool,
        cfg: &Config,
        vb: VarBuilder,

View on GitHub (pinned to d5fee525bf)

Solutions

1. Rebuild with the feature enabled: `cargo build --release --features candle-transformers/flash-attn` (requires a CUDA-capable environment).
2. Set `config.use_flash_attn = false` in code or edit the loaded config so the non-flash attention path is taken instead.
3. Run on a CUDA machine; the `candle-flash-attn` backend only supports CUDA, so on CPU/macOS the feature cannot be used.

Example fix

// before
let mut config = Config::from_json(json)?;
config.use_flash_attn = true;
// after
let mut config = Config::from_json(json)?;
// build with: cargo run --release --features candle-transformers/flash-attn
config.use_flash_attn = std::env::var("USE_FLASH_ATTN").is_ok();

#[cfg(not(feature = "flash-attn"))]
fn assert_flash_attn_unsupported() {
    assert!(!config.use_flash_attn, "use_flash_attn=true requires building with --features candle-transformers/flash-attn (CUDA only)");
}

fn flash_attn_available(cfg: &gemma2::Config) -> bool {
    !cfg.use_flash_attn || cfg!(feature = "flash-attn")
}

// Rust panics here, not Result errors; guard before constructing:
if config.use_flash_attn && !cfg!(feature = "flash-attn") {
    config.use_flash_attn = false;
}

When it happens

Trigger: Running a Gemma2 model with `use_flash_attn: true` in the model Config (e.g. loaded from a checkpoint whose config.json has flash-attention enabled) while the crate was built without `--features flash-attn`.

Common situations: Users load a Hugging Face checkpoint whose config enables flash attention, or copy example code that sets `config.use_flash_attn = true`, but run the default cargo build where the `flash-attn` feature is off (it is also unavailable on non-CUDA platforms).

Related errors

• compile with '--features flash-attn'
• compile with '--features flash-attn'
• compile with '--features flash-attn'
• compile with '--features flash-attn'
• compile with '--features flash-attn'

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/ca1b9495f7aade36. Report an issue: GitHub.