{"record":{"id":"fc724e8675da2876","repo":"huggingface/candle","slug":"the-given-quantized-dtype-is-not-supported-fo","errorCode":null,"errorMessage":"The given quantized dtype {:?} is not supported for indexed_moe_forward!","messagePattern":"The given quantized dtype (.+?) is not supported for indexed_moe_forward!","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-core/src/quantized/cuda.rs","lineNumber":617,"sourceCode":"                | GgmlDType::Q3K\n                | GgmlDType::Q4K\n                | GgmlDType::Q5K\n                | GgmlDType::Q6K\n        ) {\n            let input_storage = input.as_cuda_slice::<f32>()?;\n            let ids_storage = ids.as_cuda_slice::<u32>()?;\n            indexed_moe_forward_fused_q8_1_input(\n                &self.data.inner.slice(0..),\n                self_shape, //[num_experts, n, k]\n                self.dtype(),\n                input_storage,\n                input_l.shape(), //[batch, topk or 1, k]\n                &ids_storage.slice(0..),\n                ids_l.shape(), //[batch, topk]\n                &self.device,\n            )\n        } else {\n            crate::bail!(\n                \"The given quantized dtype {:?} is not supported for indexed_moe_forward!\",\n                self.dtype()\n            );\n        }\n    }\n\n    pub fn zeros(device: &CudaDevice, el_count: usize, dtype: GgmlDType) -> Result<Self> {\n        let size_in_bytes = ceil_div(el_count, dtype.block_size()) * dtype.type_size();\n        let padded_size_in_bytes =\n            ceil_div(el_count + MATRIX_ROW_PADDING, dtype.block_size()) * dtype.type_size();\n        let inner = device.alloc_zeros::<u8>(padded_size_in_bytes)?;\n        Ok(QCudaStorage {\n            data: PaddedCudaSlice {\n                inner,\n                len: size_in_bytes,\n            },\n            device: device.clone(),\n            dtype,","sourceCodeStart":599,"sourceCodeEnd":635,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-core/src/quantized/cuda.rs#L599-L635","documentation":"indexed_moe_forward on a quantized CUDA tensor dispatches to CUDA kernels only for specific quantized dtypes (e.g. GGUF Q4K/Q6K family supported by the MoE kernels). If the tensor's GgmlDtype is not one of the supported variants, candle bails with this message naming the dtype. It is a hard capability limitation of the quantized CUDA MoE path, not a shape or device problem.","triggerScenarios":"Calling QTensor::indexed_moe_forward (or a model using quantized MoE routing, e.g. GGUF Mixtral-style layers) on CUDA with a weight tensor whose dtype is a quantized format without a MoE kernel, such as Q4_0, Q5_0, Q8_0, or a non-GGUF quantization.","commonSituations":"Loading a GGUF model whose chosen quantization variant lacks CUDA MoE kernel support; switching from a supported Q4K model to a differently quantized one; using a quantized dtype added recently on the CPU path but not yet on the CUDA MoE path.","solutions":["Requantize the model to a supported dtype (the ones handled by the if branch above the bail, e.g. the K-quant variants with MoE kernels).","Fall back to the non-indexed quantized matmul path (matmul/dequantize_matmul) which supports more dtypes.","Run the MoE layer on CPU where more quantized dtypes are supported.","Check candle's quantized/cuda.rs for the current list of supported dtypes before choosing a quantization."],"exampleFix":"// before\nlet out = qweight.indexed_moe_forward(&ids, &topk_weights, &input)?;\n// after (requantize/choose supported dtype when creating the model)\n// quantize with a supported dtype, e.g. candle quantize --dtype q4k model.gguf","handlingStrategy":"validation","validationCode":"const SUPPORTED_MOE_DTYPES: &[GgmlDType] = &[/* dtypes handled by indexed_moe_forward, e.g. Q4K, Q6K */];\nfn moe_dtype_ok(t: &QTensor) -> bool { SUPPORTED_MOE_DTYPES.contains(&t.dtype()) }\nif !moe_dtype_ok(&qweight) { /* requantize or fallback to matmul path */ }","typeGuard":"fn is_moe_supported(t: &candle_core::quantized::QTensor) -> bool {\n    use candle_core::quantized::GgmlDType::*;\n    matches!(t.dtype(), Q4K | Q5K | Q6K) // keep in sync with quantized/cuda.rs\n}","tryCatchPattern":null,"preventionTips":["Keep in sync with the supported-dtype list in candle-core/src/quantized/cuda.rs","Prefer K-quant variants known to have CUDA MoE kernels when quantizing MoE models","Unit-test MoE forward on a tiny tensor per dtype in CI"],"tags":["cuda","quantization","moe","unsupported-dtype"],"backgroundTag":"quantized-dtype-unsupported","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}