{"record":{"id":"683f831c3facd0f1","repo":"huggingface/candle","slug":"non-contiguous-layernorm-is-not-implemented","errorCode":null,"errorMessage":"Non contiguous layernorm is not implemented","messagePattern":"Non contiguous layernorm is not implemented","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-nn/src/ops.rs","lineNumber":880,"sourceCode":"        s3: &candle::MetalStorage,\n        l3: &Layout,\n    ) -> Result<(candle::MetalStorage, Shape)> {\n        use candle::backend::BackendStorage;\n        let device = s1.device();\n        let encoder = device.command_encoder()?;\n        encoder.set_label(\"layernorm\");\n        let kernels = device.kernels();\n        let name = match (s1.dtype(), s2.dtype(), s3.dtype()) {\n            (DType::F32, DType::F32, DType::F32) => \"layernorm_f32\",\n            (DType::F16, DType::F16, DType::F16) => \"layernorm_f16\",\n            (DType::BF16, DType::BF16, DType::BF16) => \"layernorm_bf16\",\n            (dt1, dt2, dt3) => {\n                candle::bail!(\"layernorm is not implemented for {dt1:?} {dt2:?} {dt3:?}\")\n            }\n        };\n\n        if !(l1.is_contiguous() && l2.is_contiguous() && l3.is_contiguous()) {\n            candle::bail!(\"Non contiguous layernorm is not implemented\");\n        }\n\n        let last_dim = l1.dims()[l1.shape().rank() - 1];\n        let elem_count = l1.shape().elem_count();\n        let output = device\n            .new_buffer_builder()\n            .with_size_for(elem_count, s1.dtype())\n            .with_label(\"layernorm\")\n            .build()?;\n        candle_metal_kernels::call_layer_norm(\n            device.metal_device(),\n            &encoder,\n            kernels,\n            name,\n            elem_count,\n            last_dim,\n            self.eps,\n            s1.buffer(),","sourceCodeStart":862,"sourceCodeEnd":898,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-nn/src/ops.rs#L862-L898","documentation":"The Metal layernorm kernel in candle-nn requires all three inputs (input tensor, alpha/scale, beta/bias) to have contiguous memory. The Metal backend's fused layernorm CustomOp only computes on dense row-major data and has no strided-layout support, so it bails instead of producing wrong results. Call layer_norm_slow for a fallback.","triggerScenarios":"Calling candle_nn::ops::layer_norm (or any op dispatching the LayerNorm CustomOp3) on a Metal device where the input tensor, alpha, or beta has a non-contiguous layout — e.g. after slice/narrow, transpose, permute, or a view that produces strides.","commonSituations":"Passing a transposed or sliced tensor from a previous op directly into layer_norm on Apple Silicon; loading weights stored as non-contiguous views; mixing broadcasting reshapes that leave the tensor strided.","solutions":["Call .contiguous() on the input tensor (and on alpha/beta if needed) before layer_norm","Use candle_nn::ops::layer_norm_slow as a drop-in fallback that handles arbitrary layouts","Reorder ops so the tensor is reshaped into a dense form (e.g. .reshape instead of narrow+cat) before normalization","Check layouts with tensor.layout().is_contiguous() in debug code to find which of the three tensors is non-contiguous"],"exampleFix":"// before\nlet x = hidden_states.transpose(1, 2)?;\nlet out = layer_norm(&x, &alpha, &beta, 1e-5)?;\n// after\nlet x = hidden_states.transpose(1, 2)?.contiguous()?;\nlet out = layer_norm(&x, &alpha, &beta, 1e-5)?;","handlingStrategy":"validation","validationCode":"fn ensure_contiguous3(x: &Tensor, a: &Tensor, b: &Tensor) -> candle::Result<(Tensor, Tensor, Tensor)> {\n    Ok((x.contiguous()?, a.contiguous()?, b.contiguous()?))\n}\n// call before: let (x, a, b) = ensure_contiguous3(&xs, &alpha, &beta)?;","typeGuard":"fn is_contiguous(t: &Tensor) -> bool { t.layout().is_contiguous() }","tryCatchPattern":"match layer_norm(&xs, &alpha, &beta, eps) {\n    Ok(y) => y,\n    Err(e) if e.to_string().contains(\"Non contiguous\") => layer_norm_slow(&xs, &alpha, &beta, eps)?,\n    Err(e) => return Err(e),\n}","preventionTips":["Call .contiguous() after transpose/permute/narrow before normalization ops","Prefer reshape over strided views when feeding custom Metal ops","Use layer_norm_slow when layout control is hard","Assert contiguity in debug builds near model entry points"],"tags":["metal","layernorm","contiguity","candle"],"backgroundTag":"non-contiguous-tensor-unsupported","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}