abiosoft/colima · error

model %q has format %q, only GGUF models are supported Try a

Error message

model %q has format %q, only GGUF models are supported
Try a GGUF version of this model (e.g., from TheBloke on HuggingFace)

What it means

ServeDockerModel only serves GGUF models: llama-server consumes a .gguf file, so modelInfo.Config.Format != "gguf" is rejected with guidance to pick a GGUF variant. The model resolved by inspect is real and local, but its artifact format (safetensors, ggml, onnx, etc.) cannot be served.

Source

Thrown at model/docker.go:203

	guest := lima.New(host.New())

	// Set defaults
	if cfg.Threads <= 0 {
		cfg.Threads = 2
	}
	if cfg.GPULayers <= 0 {
		cfg.GPULayers = 999
	}

	// Get the model info (model should already be available via EnsureDockerModel)
	modelInfo, err := InspectDockerModel(cfg.ModelName)
	if err != nil {
		return fmt.Errorf("failed to inspect model %q: %w", cfg.ModelName, err)
	}

	// Check model format - only GGUF models are supported
	if modelInfo.Config.Format != "gguf" {
		return fmt.Errorf("model %q has format %q, only GGUF models are supported\n"+
			"Try a GGUF version of this model (e.g., from TheBloke on HuggingFace)",
			cfg.ModelName, modelInfo.Config.Format)
	}

	modelHash := modelInfo.Hash()
	if modelHash == "" {
		return fmt.Errorf("could not determine hash for model %q", cfg.ModelName)
	}

	// Ensure docker-model-runner container is running (needed to find GGUF path)
	if err := ensureDockerModelRunner(guest); err != nil {
		return err
	}

	// Find the GGUF file path (handles both Docker registry and HuggingFace models)
	ggufPath, err := findGGUFPath(guest, modelHash)
	if err != nil {
		return fmt.Errorf("could not find GGUF file for model %q: %w", cfg.ModelName, err)

View on GitHub (pinned to c3a5f9184d)

Solutions

1. Pull and serve an explicit GGUF variant (e.g. TheBloke <model>-GGUF repos on HuggingFace)
2. docker model rm the non-GGUF copy, ensure the GGUF one, then serve
3. Confirm the format before serving: colima ssh -- docker model inspect <name> and check the format field
4. If you believe the model IS gguf, update Docker Model Runner/colima in case the reported format string changed

Example fix

# before
colima model serve <model>        # safetensors build -> error

# after
colima model pull huggingface.co/TheBloke/<model>-GGUF:latest
colima model serve huggingface.co/TheBloke/<model>-GGUF:latest

info, err := InspectDockerModel(name)
if err == nil && info.Config.Format != "gguf" {
    // pick the GGUF variant before attempting serve
}

func isServableGGUF(info *DockerModelInfo) bool {
    return info != nil && strings.EqualFold(info.Config.Format, "gguf")
}

if !isServableGGUF(modelInfo) {
    return fmt.Errorf("model %q has format %q, only GGUF models are supported — pull a GGUF variant (e.g. TheBloke on HuggingFace)", cfg.ModelName, modelInfo.Config.Format)
}

When it happens

Trigger: Serving an HF model that defaults to safetensors; a ggml-era model pulled by its plain name; registry tag that points at a non-GGUF build; multi-artifact repos where inspect reports the primary (non-GGUF) format.

Common situations: Users typing a popular model name without choosing the GGUF build; models pulled for other runtimes (ollama-era ggml, transformers safetensors) reused here.

Related errors

• no GGUF layer found in model manifest
• could not find GGUF file for model %q: %w
• error starting %s: %w
• error initiating container runtime: %w
• dependency check failed for %s: %w

AI-assisted analysis of abiosoft/colima@c3a5f9184d (2026-08-15). Data as JSON: /api/errors/9bbc73389c7d7c3f. Report an issue: GitHub.