What causes "only alibi is supported as a position-embedding-type" in huggingface/candle?

Set "position_embedding_type": "alibi" in the config

only alibi is supported as a position-embedding-type

Error message

only alibi is supported as a position-embedding-type

What it means

BertEncoder::new in jina_bert only supports ALiBi positional embeddings; any other cfg.position_embedding_type (e.g. absolute/rotary) is rejected at encoder construction. This keeps the implementation scoped to the Jina BERT ALiBi variant.

Source

Thrown at candle-transformers/src/models/jina_bert.rs:350

            .take(n_heads)
            .cloned()
            .collect::<Vec<f32>>()
    };
    let slopes = Tensor::new(slopes, &Device::Cpu)?.reshape((1, (), 1, 1))?;
    alibi_bias.to_dtype(DType::F32)?.broadcast_mul(&slopes)
}

#[derive(Clone, Debug)]
struct BertEncoder {
    alibi: Tensor,
    layers: Vec<BertLayer>,
    span: tracing::Span,
}

impl BertEncoder {
    fn new(vb: VarBuilder, cfg: &Config) -> Result<Self> {
        if cfg.position_embedding_type != PositionEmbeddingType::Alibi {
            candle::bail!("only alibi is supported as a position-embedding-type")
        }
        let layers = (0..cfg.num_hidden_layers)
            .map(|index| BertLayer::new(vb.pp(format!("layer.{index}")), cfg))
            .collect::<Result<Vec<_>>>()?;
        let span = tracing::span!(tracing::Level::TRACE, "encoder");
        let alibi = build_alibi_bias(cfg)?.to_device(vb.device())?;
        Ok(Self {
            alibi,
            layers,
            span,
        })
    }
}

impl Module for BertEncoder {
    fn forward(&self, xs: &Tensor) -> Result<Tensor> {
        let _enter = self.span.enter();
        let seq_len = xs.dim(1)?;

View on GitHub (pinned to d5fee525bf)

Solutions

Set "position_embedding_type": "alibi" in the config
Use the candle bert model implementation instead for standard BERT checkpoints
Confirm you're loading an actual Jina BERT (jinaai/jina-bert-*) ALiBi checkpoint

Example fix

// before (config.json)
{"position_embedding_type": "absolute", ...}
// after
{"position_embedding_type": "alibi", ...}

Defensive patterns

Strategy: validation

Validate before calling

if cfg.position_embedding_type != jina_bert::PositionEmbeddingType::Alibi {
    return Err("jina_bert requires position_embedding_type == alibi");
}

Type guard

fn uses_alibi(c: &jina_bert::Config) -> bool {
    c.position_embedding_type == jina_bert::PositionEmbeddingType::Alibi
}

Try / catch

match jina_bert::BertModel::new(&vb, cfg) {
    Err(e) if e.to_string().contains("only alibi") =>
        Err(anyhow!("use candle's bert impl for non-ALiBi checkpoints")),
    r => r.map_err(Into::into),
}

Prevention

Only load jinaai/jina-bert-* (ALiBi) checkpoints with this model
Standard BERT checkpoints go through candle's bert module instead
Check position_embedding_type in config.json before constructing the encoder

When it happens

Trigger: Instantiating a Jina BERT model with a Config whose position_embedding_type != PositionEmbeddingType::Alibi — e.g. loading a standard BERT config.json lacking "position_embedding_type": "alibi".

Common situations: Feeding a vanilla BERT checkpoint/config into the jina_bert model code; config.json missing the field so it defaults to non-Alibi; using jina_bert for a different Jina model version.

Related errors

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/a1099617c35343af. Report an issue: GitHub.