{"record":{"id":"ae91cddb12fc9ad7","repo":"huggingface/candle","slug":"layernorm-is-not-implemented-for-dt1-dt2","errorCode":null,"errorMessage":"layernorm is not implemented for {dt1:?} {dt2:?} {dt3:?}","messagePattern":"layernorm is not implemented for (.+?) (.+?) (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-nn/src/ops.rs","lineNumber":875,"sourceCode":"        &self,\n        s1: &candle::MetalStorage,\n        l1: &Layout,\n        s2: &candle::MetalStorage,\n        l2: &Layout,\n        s3: &candle::MetalStorage,\n        l3: &Layout,\n    ) -> Result<(candle::MetalStorage, Shape)> {\n        use candle::backend::BackendStorage;\n        let device = s1.device();\n        let encoder = device.command_encoder()?;\n        encoder.set_label(\"layernorm\");\n        let kernels = device.kernels();\n        let name = match (s1.dtype(), s2.dtype(), s3.dtype()) {\n            (DType::F32, DType::F32, DType::F32) => \"layernorm_f32\",\n            (DType::F16, DType::F16, DType::F16) => \"layernorm_f16\",\n            (DType::BF16, DType::BF16, DType::BF16) => \"layernorm_bf16\",\n            (dt1, dt2, dt3) => {\n                candle::bail!(\"layernorm is not implemented for {dt1:?} {dt2:?} {dt3:?}\")\n            }\n        };\n\n        if !(l1.is_contiguous() && l2.is_contiguous() && l3.is_contiguous()) {\n            candle::bail!(\"Non contiguous layernorm is not implemented\");\n        }\n\n        let last_dim = l1.dims()[l1.shape().rank() - 1];\n        let elem_count = l1.shape().elem_count();\n        let output = device\n            .new_buffer_builder()\n            .with_size_for(elem_count, s1.dtype())\n            .with_label(\"layernorm\")\n            .build()?;\n        candle_metal_kernels::call_layer_norm(\n            device.metal_device(),\n            &encoder,\n            kernels,","sourceCodeStart":857,"sourceCodeEnd":893,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-nn/src/ops.rs#L857-L893","documentation":"The Metal layernorm kernel is only implemented for matching F32×3, F16×3 and BF16×3 dtype triples across (input, alpha, beta); any other combination bails listing the three dtypes. Mixed-precision layernorm on Metal is not auto-promoted.","triggerScenarios":"Calling the Metal layernorm op where input, alpha, and beta dtypes differ — e.g. F32 norm weights against F16 activations, or F64/integer tensors — through `xs.apply_op3_no_bwd` / the layer_norm custom op on a Metal device.","commonSituations":"Creating norm weights with default F32 while the model runs in F16/BF16 on Apple Silicon; mixing checkpoints of different precisions; numeric-experiment code using F64.","solutions":["Cast alpha and beta to the input's dtype with `.to_dtype(xs.dtype())` before the op.","Construct model weights in the model's dtype (pass `DType` when creating tensors).","Compute the norm manually with upcast ops (as `rms_norm_slow` does) if mixed precision is genuinely required."],"exampleFix":"// before: xs F16, alpha/beta F32 on Metal\nlet out = layer_norm(&xs, &alpha, &beta, eps)?;\n// after\nlet (alpha, beta) = (alpha.to_dtype(DType::F16)?, beta.to_dtype(DType::F16)?);\nlet out = layer_norm(&xs, &alpha, &beta, eps)?;","handlingStrategy":"validation","validationCode":"// before calling the Metal layernorm op\nlet dt = xs.dtype();\nlet (alpha, beta) = if alpha.dtype() != dt || beta.dtype() != dt {\n    (alpha.to_dtype(dt)?, beta.to_dtype(dt)?)\n} else {\n    (alpha.clone(), beta.clone())\n};\nlet out = layer_norm_metal(&xs, &alpha, &beta, eps)?;","typeGuard":"fn dtypes_match_norm3(x: &candle_core::Tensor, a: &candle_core::Tensor, b: &candle_core::Tensor) -> bool {\n    use candle_core::DType::*;\n    matches!(x.dtype(), F32 | F16 | BF16)\n        && a.dtype() == x.dtype()\n        && b.dtype() == x.dtype()\n}","tryCatchPattern":"match layer_norm_metal(&xs, &alpha, &beta, eps) {\n    Ok(out) => out,\n    Err(e) if e.to_string().contains(\"layernorm is not implemented for\") => {\n        let dt = xs.dtype();\n        layer_norm_metal(&xs, &alpha.to_dtype(dt)?, &beta.to_dtype(dt)?, eps)?\n    }\n    Err(e) => return Err(e),\n}","preventionTips":["Initialize norm weights with the same dtype as the model on Metal.","Run a dtype-consistency check across all model weights once after loading.","Prefer BF16/F16 checkpoints with weights already in matching precision rather than mixing F32 norms."],"tags":["metal","dtype","layernorm","mixed-precision"],"backgroundTag":"unsupported-dtype","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}