What causes "beta has to be contiguous" in huggingface/candle?

Call `.contiguous()` on the beta tensor before the op.

beta has to be contiguous

Error message

beta has to be contiguous

What it means

The `beta` (bias/shift) tensor passed to the CPU layernorm/rmsnorm-with-beta kernel has a non-contiguous layout: `beta_layout.contiguous_offsets()` returned None. Like input and alpha, beta must be a contiguous view so the kernel can index it as a flat slice per row.

Source

Thrown at candle-nn/src/ops.rs:732

        >(
            src: &[T],
            layout: &Layout,
            alpha: &[T],
            alpha_layout: &Layout,
            beta: &[T],
            beta_layout: &Layout,
            eps: f32,
        ) -> Result<(CpuStorage, Shape)> {
            let src = match layout.contiguous_offsets() {
                None => candle::bail!("input has to be contiguous"),
                Some((o1, o2)) => &src[o1..o2],
            };
            let alpha = match alpha_layout.contiguous_offsets() {
                None => candle::bail!("alpha has to be contiguous"),
                Some((o1, o2)) => &alpha[o1..o2],
            };
            let beta = match beta_layout.contiguous_offsets() {
                None => candle::bail!("beta has to be contiguous"),
                Some((o1, o2)) => &beta[o1..o2],
            };
            let el_count = layout.shape().elem_count();
            let dims = layout.shape().dims();
            let dim_m1 = dims[dims.len() - 1];
            let mut dst = vec![T::zero(); el_count];
            src.par_chunks(dim_m1)
                .zip(dst.par_chunks_mut(dim_m1))
                .for_each(|(src, dst)| {
                    let mut sum = 0f32;
                    let mut sum2 = 0f32;
                    for v in src {
                        let v = v.as_();
                        sum += v;
                        sum2 += v * v;
                    }
                    let mean = sum / dim_m1 as f32;
                    let var = sum2 / dim_m1 as f32 - mean * mean;

View on GitHub (pinned to d5fee525bf)

Solutions

Call `.contiguous()` on the beta tensor before the op.
Store biases as their own 1-D contiguous tensors instead of views into larger buffers.
Verify weight-conversion code preserves contiguity for norm parameters.

Example fix

// before
let beta = fused.narrow(0, bias_off, h)?;
let out = layer_norm(&x, &alpha, &beta, eps)?;
// after
let beta = fused.narrow(0, bias_off, h)?.contiguous()?;
let out = layer_norm(&x, &alpha, &beta, eps)?;

Defensive patterns

Strategy: validation

Validate before calling

// before calling the op
if !beta.layout().is_contiguous() {
    let beta = beta.contiguous()?;
}
let out = layer_norm(&x, &alpha, &beta, eps)?;

Type guard

fn is_valid_beta(t: &candle_core::Tensor) -> bool {
    t.dims().len() == 1 && t.layout().is_contiguous()
}

Try / catch

match layer_norm(&x, &alpha, &beta, eps) {
    Ok(out) => out,
    Err(e) if e.to_string().contains("beta has to be contiguous") => layer_norm(&x, &alpha, &beta.contiguous()?, eps)?,
    Err(e) => return Err(e),
}

Prevention

Store norm biases as standalone contiguous 1-D tensors.
Add a one-time contiguity check when loading weights rather than per forward pass.
Avoid permute/narrow on bias tensors during model surgery.

When it happens

Trigger: Passing a beta tensor derived from transpose/slice/narrow of another tensor into the CPU layernorm op, e.g. `bias.permute(...)` or a column slice of a fused weight matrix.

Common situations: Extracting norm biases from a fused QKV/norm parameter block with strided slicing; converting weights between layouts during quantization or model surgery.

Related errors

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/02979835315690a3. Report an issue: GitHub.