huggingface/candle · error

upcasting is not supported {:?}

Error message

upcasting is not supported {:?}

What it means

GPT-BigCode attention upcasts query/key/value to f32 for scaling before the matmul; this implementation only supports f32 inputs and refuses any other dtype with this error. The comment notes upcasting for f16 was not implemented in this port.

Source

Thrown at candle-transformers/src/models/bigcode.rs:167

            use_cache: cfg.use_cache,
            kv_dim,
            head_dim,
            num_heads: cfg.num_attention_heads,
            multi_query: cfg.multi_query,
        })
    }

    fn attn(
        &self,
        query: &Tensor,
        key: &Tensor,
        value: &Tensor,
        attention_mask: &Tensor,
    ) -> Result<Tensor> {
        if query.dtype() != DType::F32 {
            // If we start supporting f16 models, we may need the upcasting scaling bits.
            // https://github.com/huggingface/transformers/blob/a0042379269bea9182c1f87e6b2eee4ba4c8cce8/src/transformers/models/gpt_bigcode/modeling_gpt_bigcode.py#L133
            candle::bail!("upcasting is not supported {:?}", query.dtype())
        }
        let scale_factor = 1f64 / (self.head_dim as f64).sqrt();
        let initial_query_shape = query.shape();
        let key_len = key.dim(D::Minus1)?;
        let (query, key, attn_shape, attn_view) = if self.multi_query {
            let (b_sz, query_len, _) = query.dims3()?;
            let query = query.reshape((b_sz, query_len * self.num_heads, self.head_dim))?;
            let attn_shape = (b_sz, query_len, self.num_heads, key_len);
            let attn_view = (b_sz, query_len * self.num_heads, key_len);
            (query, key.clone(), attn_shape, attn_view)
        } else {
            let (b_sz, _num_heads, query_len, _head_dim) = query.dims4()?;
            let query = query.reshape((b_sz, query_len * self.num_heads, self.head_dim))?;
            let key = key.reshape((b_sz * self.num_heads, self.head_dim, key_len))?;
            let attn_shape = (b_sz, self.num_heads, query_len, key_len);
            let attn_view = (b_sz * self.num_heads, query_len, key_len);
            (query, key, attn_shape, attn_view)
        };

View on GitHub (pinned to d5fee525bf)

Solutions

1. Convert the model to f32 before inference: model.to_dtype(DType::F32) or to_dtype on device-loaded varmap
2. Load the checkpoint with .to_dtype(DType::F32) when reading safetensors
3. Run the model on a device with enough memory for f32, or patch the attention to upcast internally
4. Use a different model implementation that supports f16 if memory is the constraint

Example fix

// before
let model = GPTBigCode::load(...)?;  // weights in f16
// after
let model = GPTBigCode::load(...)?;
let model = model.to_dtype(&device, DType::F32)?;

if model.dtype() != DType::F32 {
    model.to_dtype(&device, DType::F32)?;
}

fn is_f32(t: &Tensor) -> bool { t.dtype() == DType::F32 }

match model.forward(&input_ids, 0) {
    Ok(v) => v,
    Err(e) if e.to_string().contains("upcasting is not supported") => {
        let model = model.to_dtype(&device, DType::F32)?;
        model.forward(&input_ids, 0)
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: Running the bigcode model with weights loaded in f16 (or bf16): attn is called with query.dtype() != F32 during forward pass.

Common situations: Loading safetensors checkpoints that are stored in half precision without converting, or using .to_dtype(DType::F16) for memory savings before inference.

Related errors

• the accelerate backend does not support f16 matmul
• attribute {} of type TENSOR was an invalid data_type number
• attribute {} of type TENSOR has an unsupported data_type {}
• unsupported 'value' data-type {dt:?} for {name}
• Qwen3 does not support f64; load weights as f32 or bf16 (CPU

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/d0e026b48d49cab3. Report an issue: GitHub.