huggingface/candle · error

The given quantized dtype {:?} is not supported for indexed_

Error message

The given quantized dtype {:?} is not supported for indexed_moe_forward!

What it means

indexed_moe_forward on a quantized CUDA tensor dispatches to CUDA kernels only for specific quantized dtypes (e.g. GGUF Q4K/Q6K family supported by the MoE kernels). If the tensor's GgmlDtype is not one of the supported variants, candle bails with this message naming the dtype. It is a hard capability limitation of the quantized CUDA MoE path, not a shape or device problem.

Source

Thrown at candle-core/src/quantized/cuda.rs:617

                | GgmlDType::Q3K
                | GgmlDType::Q4K
                | GgmlDType::Q5K
                | GgmlDType::Q6K
        ) {
            let input_storage = input.as_cuda_slice::<f32>()?;
            let ids_storage = ids.as_cuda_slice::<u32>()?;
            indexed_moe_forward_fused_q8_1_input(
                &self.data.inner.slice(0..),
                self_shape, //[num_experts, n, k]
                self.dtype(),
                input_storage,
                input_l.shape(), //[batch, topk or 1, k]
                &ids_storage.slice(0..),
                ids_l.shape(), //[batch, topk]
                &self.device,
            )
        } else {
            crate::bail!(
                "The given quantized dtype {:?} is not supported for indexed_moe_forward!",
                self.dtype()
            );
        }
    }

    pub fn zeros(device: &CudaDevice, el_count: usize, dtype: GgmlDType) -> Result<Self> {
        let size_in_bytes = ceil_div(el_count, dtype.block_size()) * dtype.type_size();
        let padded_size_in_bytes =
            ceil_div(el_count + MATRIX_ROW_PADDING, dtype.block_size()) * dtype.type_size();
        let inner = device.alloc_zeros::<u8>(padded_size_in_bytes)?;
        Ok(QCudaStorage {
            data: PaddedCudaSlice {
                inner,
                len: size_in_bytes,
            },
            device: device.clone(),
            dtype,

View on GitHub (pinned to d5fee525bf)

Solutions

1. Requantize the model to a supported dtype (the ones handled by the if branch above the bail, e.g. the K-quant variants with MoE kernels).
2. Fall back to the non-indexed quantized matmul path (matmul/dequantize_matmul) which supports more dtypes.
3. Run the MoE layer on CPU where more quantized dtypes are supported.
4. Check candle's quantized/cuda.rs for the current list of supported dtypes before choosing a quantization.

Example fix

// before
let out = qweight.indexed_moe_forward(&ids, &topk_weights, &input)?;
// after (requantize/choose supported dtype when creating the model)
// quantize with a supported dtype, e.g. candle quantize --dtype q4k model.gguf

const SUPPORTED_MOE_DTYPES: &[GgmlDType] = &[/* dtypes handled by indexed_moe_forward, e.g. Q4K, Q6K */];
fn moe_dtype_ok(t: &QTensor) -> bool { SUPPORTED_MOE_DTYPES.contains(&t.dtype()) }
if !moe_dtype_ok(&qweight) { /* requantize or fallback to matmul path */ }

fn is_moe_supported(t: &candle_core::quantized::QTensor) -> bool {
    use candle_core::quantized::GgmlDType::*;
    matches!(t.dtype(), Q4K | Q5K | Q6K) // keep in sync with quantized/cuda.rs
}

When it happens

Trigger: Calling QTensor::indexed_moe_forward (or a model using quantized MoE routing, e.g. GGUF Mixtral-style layers) on CUDA with a weight tensor whose dtype is a quantized format without a MoE kernel, such as Q4_0, Q5_0, Q8_0, or a non-GGUF quantization.

Common situations: Loading a GGUF model whose chosen quantization variant lacks CUDA MoE kernel support; switching from a supported Q4K model to a differently quantized one; using a quantized dtype added recently on the CPU path but not yet on the CUDA MoE path.

Related errors

• only f32 can be quantized
• quantized embedding requires contiguous ids
• quantized embedding hidden size {hidden} is not divisible by
• quantized tensor has {} bytes, expected {expected_size}
• unexpected rhs shape in dmmv {:?}

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/fc724e8675da2876. Report an issue: GitHub.