What causes "compile with '--features flash-attn'" in huggingface/candle?

Build with the feature: `cargo run --release --features candle-transformers/flash-attn` on a CUDA machine.

compile with '--features flash-attn'

Error message

compile with '--features flash-attn'

What it means

llama.rs's `flash_attn` helper is replaced by an `unimplemented!` stub when compiled without the `flash-attn` cargo feature. If flash attention is requested in the config, `CausalSelfAttention::forward` calls the stub and the run panics.

Source

Thrown at candle-transformers/src/models/llama.rs:258

    span: tracing::Span,
    span_rot: tracing::Span,
    max_position_embeddings: usize,
}

#[cfg(feature = "flash-attn")]
fn flash_attn(
    q: &Tensor,
    k: &Tensor,
    v: &Tensor,
    softmax_scale: f32,
    causal: bool,
) -> Result<Tensor> {
    candle_flash_attn::flash_attn(q, k, v, softmax_scale, causal)
}

#[cfg(not(feature = "flash-attn"))]
fn flash_attn(_: &Tensor, _: &Tensor, _: &Tensor, _: f32, _: bool) -> Result<Tensor> {
    unimplemented!("compile with '--features flash-attn'")
}

impl CausalSelfAttention {
    fn apply_rotary_emb(&self, x: &Tensor, index_pos: usize, cache: &Cache) -> Result<Tensor> {
        let _enter = self.span_rot.enter();
        let (_b_sz, _, seq_len, _hidden_size) = x.dims4()?;
        let cos = cache.cos.narrow(0, index_pos, seq_len)?;
        let sin = cache.sin.narrow(0, index_pos, seq_len)?;
        candle_nn::rotary_emb::rope(x, &cos, &sin)
    }

    fn forward(
        &self,
        x: &Tensor,
        index_pos: usize,
        block_idx: usize,
        cache: &mut Cache,
    ) -> Result<Tensor> {

View on GitHub (pinned to d5fee525bf)

Solutions

Build with the feature: `cargo run --release --features candle-transformers/flash-attn` on a CUDA machine.
Set `config.use_flash_attn = false` before instantiating the model.
If porting example code, remove the flash-attn flag unless your build and hardware support it.

Example fix

// before
let mut config = llama::Config::from_json(json)?;
config.use_flash_attn = true;
// after
let mut config = llama::Config::from_json(json)?;
config.use_flash_attn = false; // or build with --features candle-transformers/flash-attn

Defensive patterns

Strategy: validation

Validate before calling

let mut config = llama::Config::from_json(json)?;
if config.use_flash_attn && !cfg!(feature = "flash-attn") {
    config.use_flash_attn = false;
}

Type guard

fn flash_attention_ok(cfg: &llama::Config) -> bool {
    !cfg.use_flash_attn || cfg!(feature = "flash-attn")
}

Try / catch

assert!(flash_attention_ok(&config), "rebuild with --features candle-transformers/flash-attn or set use_flash_attn=false");

Prevention

Llama 3 configs may enable flash attention; always inspect after loading
Remember the feature needs CUDA; disable on CPU/macOS
Centralize config sanitization for all models in the binary
Add a unit test asserting Model::new succeeds without the feature

When it happens

Trigger: Running a LLaMA model with `use_flash_attn: true` in Config (common when reusing llama3/3.1 checkpoint configs) without `--features flash-attn`.

Common situations: Checkpoint config.json enables flash attention; default builds; non-CUDA platforms (macOS, CPU-only Linux) where candle-flash-attn cannot be enabled.

Related errors

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/cadf00cc7d9be7af. Report an issue: GitHub.