aaif-goose/goose · error

llama.cpp model '{}' is missing a quantization

Error message

llama.cpp model '{}' is missing a quantization

What it means

When a download request selects the llamacpp backend, local_model_id_from_request() requires a quantization: either selection.variant_id or a spec that parses as 'owner/repo:QUANT'. With neither present it bails with this error. The quantization is needed because llama.cpp model ids embed the quant (model_id_from_repo builds 'owner/repo:QUANT'). MLX selections don't need one.

Source

Thrown at crates/goose-local-inference/src/management.rs:707

                variant_id,
            })),
            _ => anyhow::bail!("Unknown local inference backend '{}'", backend_id),
        }
    } else {
        Ok(None)
    }
}

async fn local_model_id_from_request(
    req: &LocalInferenceModelDownloadRequest,
    selection: Option<&LocalModelSelection>,
) -> Result<String> {
    if let Some(selection) = selection {
        return match selection.backend_id.as_str() {
            "mlx" => Ok(selection.repo_id.clone()),
            "llamacpp" => {
                let quantization = selection.variant_id.as_deref().ok_or_else(|| {
                    anyhow!(
                        "llama.cpp model '{}' is missing a quantization",
                        selection.repo_id
                    )
                })?;
                Ok(model_id_from_repo(&selection.repo_id, quantization))
            }
            _ => anyhow::bail!("Unknown local inference backend '{}'", selection.backend_id),
        };
    }

    if let Ok((repo_id, quantization)) = hf_models::parse_model_spec(&req.spec) {
        return Ok(model_id_from_repo(&repo_id, &quantization));
    }

    let variants = hf_models::get_repo_local_variants(&req.spec).await?;
    let has_llamacpp = variants
        .iter()
        .any(|variant| variant.backend_id == "llamacpp");

View on GitHub (pinned to 3810898a74)

Solutions

1. Qualify the spec with a quantization: "owner/repo:Q4_K_M" (Q4_K_M, Q5_K_M, Q8_0, etc.).
2. Or pass variant_id explicitly (e.g. "Q4_K_M") alongside backend_id="llamacpp".
3. Or drop backend_id and let auto-detection resolve the variant from the repo's GGUF files.
4. Choose "mlx" instead if you want to download by bare repo id.

Example fix

// before
let req = DownloadRequest { spec: "Qwen/Qwen2.5-7B".into(), backend_id: Some("llamacpp".into()), variant_id: None, ..Default::default() };
// -> llama.cpp model 'Qwen/Qwen2.5-7B' is missing a quantization

// after
let req = DownloadRequest { spec: "Qwen/Qwen2.5-7B:Q4_K_M".into(), backend_id: Some("llamacpp".into()), variant_id: None, ..Default::default() };

if req.backend_id.as_deref() == Some("llamacpp") && req.variant_id.is_none() {
    if let Ok((_, quant)) = hf_models::parse_model_spec(&req.spec) {
        req.variant_id = Some(quant);
    } else {
        anyhow::bail!("llamacpp downloads need a quantization, e.g. 'owner/repo:Q4_K_M'");
    }
}
let id = resolve_model_id(&req).await?;

fn has_llamacpp_quantization(spec: &str, variant_id: Option<&str>) -> bool {
    variant_id.is_some() || hf_models::parse_model_spec(spec).is_ok()
}

When it happens

Trigger: backend_id="llamacpp" with spec="Qwen/Qwen2.5-7B" (no :Q4_K_M suffix) and variant_id=None. Also when the spec's suffix is not parseable by parse_model_spec, so variant_id stays None.

Common situations: Users copying the bare HuggingFace repo name; UI sending the repo id in spec and forgetting the variant picker; prompting an LLM agent that fills spec without the quant suffix.

Related errors

• Unknown local inference backend '{}'
• Model spec '{}' is ambiguous; choose one of: {}
• llama.cpp model '{}' is missing a quantization
• MLX model {} has no downloadable files
• Local inference with the bundled llama.cpp backend requires

AI-assisted analysis of aaif-goose/goose@3810898a74 (2026-08-16). Data as JSON: /api/errors/b0613670b09f19cb. Report an issue: GitHub.