unslothai/unsloth · error · HTTPException

gguf_error

Error message

gguf_error

What it means

HTTP 409 from the embedding-model endpoint when the GGUF backend probe returns an error string. gguf_error is first computed by _local_gguf_backend_error(model) (always), and if that is None and the server is not local-only-load, by _hf_gguf_backend_error(model, hf_token) — a Hub probe that is intentionally skipped offline because list_repo_files can hang. A non-empty gguf_error means the model cannot serve as the GGUF-backed embedder on the active llama-server backend; the detail is the specific probe message (missing .gguf files, wrong layout, unreachable repo, ...).

Source

Thrown at studio/backend/routes/settings.py:1658

            # Require a genuinely loadable cache (config + weights), not just a resolved refs/main,
            # so a metadata-only partial cache still gets the forceable 409.
            offline_cached = local_only_load and hf_cache_snapshot_is_loadable(model)
            if not offline_cached:
                raise HTTPException(
                    status_code = 409,
                    detail = (
                        f"Could not verify {model!r} as an embedding model on "
                        "Hugging Face (it may be the wrong model type, gated, or "
                        "you may be offline)."
                    ),
                )
        # The Hub GGUF probe (list_repo_files) can hang offline; skip it. Local check stays.
        gguf_error = _local_gguf_backend_error(model)
        if gguf_error is None and not local_only_load:
            gguf_error = _hf_gguf_backend_error(model, hf_token)
        if gguf_error:
            raise HTTPException(status_code = 409, detail = gguf_error)
    set_rag_embedding_model(model)
    logger.info(
        "settings.embedding_model_updated subject=%s model=%s forced=%s",
        current_subject,
        model,
        payload.force,
    )
    return _embedding_model_response()


@router.delete("/embedding-model", response_model = EmbeddingModelResponse)
def reset_embedding_model(
    current_subject: str = Depends(get_current_subject),
) -> EmbeddingModelResponse:
    """Clear the override, returning to the env/default model."""
    reset_rag_embedding_model()
    logger.info("settings.embedding_model_reset subject=%s", current_subject)
    return _embedding_model_response()

View on GitHub (pinned to 203007d190)

Solutions

1. Read the detail string — it distinguishes 'no local GGUF found' from Hub-side failures.
2. Verify the repo really hosts .gguf files (check the Files tab) and that at least one is fully downloaded locally.
3. For gated GGUF repos, set a token with access via the token endpoint and retry while online.
4. If the repo has no GGUF variant, either switch to a safetensors sentence-transformers model or change the backend so the GGUF path is not required.

Example fix

# before
PUT /settings/embedding-model {"model": "user/mixed-model"}  # 409: no .gguf weights

# after
# confirm GGUF exists on the Hub, e.g. user/mixed-model-GGUF
huggingface-cli download user/mixed-model-GGUF --include "*.gguf"
PUT /settings/embedding-model {"model": "user/mixed-model-GGUF"}

const localGgufs = await listLocalCacheFiles(model).then(fs => fs.filter(f => f.endsWith('.gguf')));
if (localGgufs.length === 0 && !navigator.onLine) {
  throw new Error('No local .gguf files and offline; cannot use as GGUF embedder');
}
await api.put('/settings/embedding-model', { model });

function hasGgufWeights(files: string[]): boolean {
  return files.some(f => f.toLowerCase().endsWith('.gguf'));
}

try { await api.put('/settings/embedding-model', { model }); }
catch (e) {
  if (e.status === 409 && /gguf/i.test(e.detail)) { showGgufHelp(e.detail); return; }
  throw e;
}

When it happens

Trigger: PUT embedding-model naming a GGUF repo while the llama-server backend is active, where the repo has no usable .gguf files locally (local probe fails), or — when online — the Hub probe finds no downloadable GGUF weights or the repo is inaccessible with the supplied token.

Common situations: Pointing the embedder at a GGUF-quantized repo that actually ships only safetensors; partial local GGUF cache; gated GGUF repos without a token; typos in the repo id surfacing as probe errors online.

Related errors

• Could not verify {model!r} as an embedding model on Hugging
• {model!r} has cached pickle weights that cannot be security-
• {model!r} is flagged as unsafe by Hugging Face's security sc
• MLXInferenceBackend cannot load GGUF model '{model_name}': G
• gguf_variant is required when export_type is 'gguf'

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/b24ced3b3f4526e1. Report an issue: GitHub.