{"record":{"id":"417e00bc0accba9f","repo":"huggingface/candle","slug":"shape-mismatch-in-layer-norm-src-alpha","errorCode":null,"errorMessage":"shape mismatch in layer-norm src: {:?} alpha: {:?} beta: {:?}","messagePattern":"shape mismatch in layer-norm src: (.+?) alpha: (.+?) beta: (.+?)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-nn/src/ops.rs","lineNumber":937,"sourceCode":"    let x = x.to_dtype(internal_dtype)?;\n    let x = {\n        let mean_x = (x.sum_keepdim(D::Minus1)? / hidden_size as f64)?;\n        x.broadcast_sub(&mean_x)?\n    };\n    let norm_x = (x.sqr()?.sum_keepdim(D::Minus1)? / hidden_size as f64)?;\n    let x_normed = x.broadcast_div(&(norm_x + eps as f64)?.sqrt()?)?;\n    x_normed\n        .to_dtype(x_dtype)?\n        .broadcast_mul(alpha)?\n        .broadcast_add(beta)\n}\n\npub fn layer_norm(xs: &Tensor, alpha: &Tensor, beta: &Tensor, eps: f32) -> Result<Tensor> {\n    let hidden_size_xs = xs.dim(D::Minus1)?;\n    let hidden_size_alpha = alpha.dims1()?;\n    let hidden_size_beta = beta.dims1()?;\n    if hidden_size_xs != hidden_size_alpha || hidden_size_xs != hidden_size_beta {\n        candle::bail!(\n            \"shape mismatch in layer-norm src: {:?} alpha: {:?} beta: {:?}\",\n            xs.shape(),\n            alpha.shape(),\n            beta.shape()\n        )\n    }\n    xs.apply_op3_no_bwd(alpha, beta, &LayerNorm { eps })\n}\n\n// https://pytorch.org/docs/stable/generated/torch.nn.PixelShuffle.html\npub fn pixel_shuffle(xs: &Tensor, upscale_factor: usize) -> Result<Tensor> {\n    let (b_size, c, h, w) = xs.dims4()?;\n    let out_c = c / upscale_factor / upscale_factor;\n    xs.reshape((b_size, out_c, upscale_factor, upscale_factor, h, w))?\n        .permute((0, 1, 4, 2, 5, 3))?\n        .reshape((b_size, out_c, h * upscale_factor, w * upscale_factor))\n}\n","sourceCodeStart":919,"sourceCodeEnd":955,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-nn/src/ops.rs#L919-L955","documentation":"layer_norm validates that the last dimension of the input tensor equals the length (dims1) of both the alpha (scale) and beta (bias) tensors. The elementwise affine parameters must span exactly the normalized (hidden) dimension; if not, the shapes are printed and an error is raised before the op is dispatched.","triggerScenarios":"Calling candle_nn::ops::layer_norm(xs, alpha, beta, eps) where xs.dim(D::Minus1) != alpha.dims1() or != beta.dims1(), e.g. weight tensors from a different hidden size than the activation tensor.","commonSituations":"Loading weights from a model checkpoint whose hidden_size differs from the model config; passing full 2D weight matrices instead of 1D vectors; mixing up layers (e.g. MLP weights fed into a norm layer); reshaping errors that change the last dim.","solutions":["Verify the last dim of xs equals alpha.len() and beta.len() before the call","Ensure alpha and beta are 1-D tensors of size hidden_size (use dims1-compatible tensors)","Check the checkpoint/config hidden_size matches the model that produced xs","Use Tensor::broadcast-agnostic reshape: keep weight as shape (hidden_size,) not (1, hidden_size)"],"exampleFix":"// before\nlet alpha = Tensor::new(vec![0.0f32; 768], &dev)?; // wrong size\nlet out = layer_norm(&xs, &alpha, &beta, 1e-5)?;\n// after\nlet hidden = xs.dim(candle::D::Minus1)?;\nlet alpha = Tensor::new(vec![0.0f32; hidden], &dev)?;\nlet out = layer_norm(&xs, &alpha, &beta, 1e-5)?;","handlingStrategy":"validation","validationCode":"fn check_layernorm_shapes(xs: &Tensor, alpha: &Tensor, beta: &Tensor) -> candle::Result<()> {\n    let h = xs.dim(candle::D::Minus1)?;\n    if alpha.dims1()? != h || beta.dims1()? != h {\n        candle::bail!(\"layer_norm shape mismatch: xs hidden={h} alpha={:?} beta={:?}\", alpha.shape(), beta.shape());\n    }\n    Ok(())\n}","typeGuard":"fn layernorm_shapes_ok(xs: &Tensor, alpha: &Tensor, beta: &Tensor) -> bool {\n    xs.dim(candle::D::Minus1).map(|h| alpha.dims1() == Ok(h) && beta.dims1() == Ok(h)).unwrap_or(false)\n}","tryCatchPattern":"match layer_norm(&xs, &alpha, &beta, eps) {\n    Ok(y) => y,\n    Err(e) if e.to_string().contains(\"shape mismatch in layer-norm\") => {\n        // log alpha/beta shapes and the expected hidden size, then re-raise with context\n        Err(e)\n    }\n    Err(e) => Err(e),\n}","preventionTips":["Keep alpha/beta as 1-D tensors of exactly hidden_size","Cross-check hidden_size between model config and checkpoint weights","Don't pass 2-D or broadcast-shaped weight matrices to layer_norm","Unit-test norm layers with the real checkpoint shapes at startup"],"tags":["shape-mismatch","layernorm","validation","candle"],"backgroundTag":"shape-mismatch","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}