mamba layers are not yet supported in GraniteMoeHybrid infer

Error message

mamba layers are not yet supported in GraniteMoeHybrid inference

What it means

GraniteMoeHybrid in candle only implements Attention blocks. When iterating layers, a layer typed Mamba hits a bail with a TODO — Mamba (SSM) blocks are not implemented, so a hybrid checkpoint containing them cannot run inference.

Source

Thrown at candle-transformers/src/models/granitemoehybrid.rs:554

        if cfg.layer_types.len() != cfg.num_hidden_layers {
            candle::bail!(
                "layer_types length {} does not match num_hidden_layers {}",
                cfg.layer_types.len(),
                cfg.num_hidden_layers
            );
        }
        let blocks = cfg
            .layer_types
            .iter()
            .enumerate()
            .map(|(idx, layer_ty)| match layer_ty {
                GraniteMoeHybridLayerType::Attention => {
                    Block::load(vb.pp(format!("model.layers.{idx}")), cfg)
                }
                GraniteMoeHybridLayerType::Mamba => {
                    // TODO: Not supprting Mamba layers (blocks) for now,
                    // so we only iterate over attention layers.
                    candle::bail!(
                        "mamba layers are not yet supported in GraniteMoeHybrid inference"
                    )
                }
            })
            .collect::<Result<Vec<_>>>()?;

        Ok(Self {
            word_token_embedding: wte,
            blocks,
            ln_f,
            logits_scale: if cfg.logits_scaling == 0.0 {
                1.0
            } else {
                1.0 / cfg.logits_scaling
            },
            embedding_scale: cfg.embedding_multiplier,
        })
    }

View on GitHub (pinned to d5fee525bf)

Solutions

Use a GraniteMoeHybrid checkpoint variant whose layer_types are all Attention
Wait for / upgrade to a candle version implementing Mamba blocks
Contribute or patch the Mamba block implementation in granitemoehybrid.rs

Example fix

// before (config.json)
"layer_types": ["attention", "mamba", "attention", ...]
// after (attention-only variant)
"layer_types": ["attention", "attention", "attention", ...]

Defensive patterns

Strategy: validation

Validate before calling

if cfg.layer_types.iter().any(|t| matches!(t, granitemoehybrid::GraniteMoeHybridLayerType::Mamba)) {
    return Err("checkpoint contains Mamba layers, unsupported by candle GraniteMoeHybrid");
}

Type guard

fn is_attention_only(c: &granitemoehybrid::GraniteMoeHybridInternalConfig) -> bool {
    c.layer_types.iter().all(|t| matches!(t, granitemoehybrid::GraniteMoeHybridLayerType::Attention))
}

Try / catch

match granitemoehybrid::Model::new(&vb, &cfg) {
    Err(e) if e.to_string().contains("mamba layers") =>
        Err(anyhow!("choose an attention-only GraniteMoeHybrid checkpoint or a different runtime")),
    r => r.map_err(Into::into),
}

Prevention

Inspect layer_types in config.json before selecting this model implementation
Use attention-only Granite hybrid checkpoints with candle
Track candle releases for Mamba support or use a runtime that implements hybrid SSM layers

When it happens

Trigger: Loading a GraniteMoeHybrid checkpoint whose layer_types includes GraniteMoeHybridLayerType::Mamba and calling load/forward.

Common situations: Using a newer/variant Granite hybrid model that mixes Mamba layers while the candle port only supports the attention-only variant; not filtering for attention-only checkpoints.

Related errors

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/0cfdfb50a3c354d0. Report an issue: GitHub.