{"record":{"id":"d0e026b48d49cab3","repo":"huggingface/candle","slug":"upcasting-is-not-supported","errorCode":null,"errorMessage":"upcasting is not supported {:?}","messagePattern":"upcasting is not supported (.+?)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-transformers/src/models/bigcode.rs","lineNumber":167,"sourceCode":"            use_cache: cfg.use_cache,\n            kv_dim,\n            head_dim,\n            num_heads: cfg.num_attention_heads,\n            multi_query: cfg.multi_query,\n        })\n    }\n\n    fn attn(\n        &self,\n        query: &Tensor,\n        key: &Tensor,\n        value: &Tensor,\n        attention_mask: &Tensor,\n    ) -> Result<Tensor> {\n        if query.dtype() != DType::F32 {\n            // If we start supporting f16 models, we may need the upcasting scaling bits.\n            // https://github.com/huggingface/transformers/blob/a0042379269bea9182c1f87e6b2eee4ba4c8cce8/src/transformers/models/gpt_bigcode/modeling_gpt_bigcode.py#L133\n            candle::bail!(\"upcasting is not supported {:?}\", query.dtype())\n        }\n        let scale_factor = 1f64 / (self.head_dim as f64).sqrt();\n        let initial_query_shape = query.shape();\n        let key_len = key.dim(D::Minus1)?;\n        let (query, key, attn_shape, attn_view) = if self.multi_query {\n            let (b_sz, query_len, _) = query.dims3()?;\n            let query = query.reshape((b_sz, query_len * self.num_heads, self.head_dim))?;\n            let attn_shape = (b_sz, query_len, self.num_heads, key_len);\n            let attn_view = (b_sz, query_len * self.num_heads, key_len);\n            (query, key.clone(), attn_shape, attn_view)\n        } else {\n            let (b_sz, _num_heads, query_len, _head_dim) = query.dims4()?;\n            let query = query.reshape((b_sz, query_len * self.num_heads, self.head_dim))?;\n            let key = key.reshape((b_sz * self.num_heads, self.head_dim, key_len))?;\n            let attn_shape = (b_sz, self.num_heads, query_len, key_len);\n            let attn_view = (b_sz * self.num_heads, query_len, key_len);\n            (query, key, attn_shape, attn_view)\n        };","sourceCodeStart":149,"sourceCodeEnd":185,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-transformers/src/models/bigcode.rs#L149-L185","documentation":"GPT-BigCode attention upcasts query/key/value to f32 for scaling before the matmul; this implementation only supports f32 inputs and refuses any other dtype with this error. The comment notes upcasting for f16 was not implemented in this port.","triggerScenarios":"Running the bigcode model with weights loaded in f16 (or bf16): attn is called with query.dtype() != F32 during forward pass.","commonSituations":"Loading safetensors checkpoints that are stored in half precision without converting, or using .to_dtype(DType::F16) for memory savings before inference.","solutions":["Convert the model to f32 before inference: model.to_dtype(DType::F32) or to_dtype on device-loaded varmap","Load the checkpoint with .to_dtype(DType::F32) when reading safetensors","Run the model on a device with enough memory for f32, or patch the attention to upcast internally","Use a different model implementation that supports f16 if memory is the constraint"],"exampleFix":"// before\nlet model = GPTBigCode::load(...)?;  // weights in f16\n// after\nlet model = GPTBigCode::load(...)?;\nlet model = model.to_dtype(&device, DType::F32)?;","handlingStrategy":"validation","validationCode":"if model.dtype() != DType::F32 {\n    model.to_dtype(&device, DType::F32)?;\n}","typeGuard":"fn is_f32(t: &Tensor) -> bool { t.dtype() == DType::F32 }","tryCatchPattern":"match model.forward(&input_ids, 0) {\n    Ok(v) => v,\n    Err(e) if e.to_string().contains(\"upcasting is not supported\") => {\n        let model = model.to_dtype(&device, DType::F32)?;\n        model.forward(&input_ids, 0)\n    }\n    Err(e) => return Err(e),\n}","preventionTips":["Load bigcode weights with .to_dtype(DType::F32) at load time","Don't cast the model to F16 for memory savings with this implementation","Check safetensors dtype before loading","Monitor memory; if f32 doesn't fit, use an implementation supporting f16"],"tags":["rust","candle","bigcode","dtype","f16"],"backgroundTag":"unsupported-dtype-f16","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}