cjpais/Handy · error · anyhow::Error

Failed to create session for whisper model

Error message

Failed to create session for whisper model {}: {}

What it means

Raised when the GGUF loaded successfully but creating an inference session (model.session()) fails. The model bytes are fine; the session layer — which allocates compute buffers, binds the backend context, and reads GGUF metadata — could not start. Common native causes are memory allocation failures (large models), backend context creation errors (GPU), or metadata the session builder cannot reconcile. A 'loading_failed' event is emitted before returning.

Solutions

1. Free memory/VRAM (close other model-using apps) and retry the load.
2. Switch the accelerator to CPU (or a different GPU device) in settings and retry.
3. Use a smaller model size/quantization for the same family.
4. Update GPU drivers (or on Linux, check the Vulkan ICD) if session creation fails only on GPU backends.
5. Retry once after a brief wait — transient device-lost states can clear.

// Rough headroom check before loading big GGUFs
let need = std::fs::metadata(&model_path)?.len() * 2; // weights + compute buffers
let avail = available_memory_bytes()?;
if avail < need {
    anyhow::bail!("need ~{} bytes for session, only {} free", need, avail);
}

match model.session() {
    Ok(s) => Ok(s),
    Err(e) if gpu_requested => {
        warn!("session on GPU failed ({e}); retrying on CPU");
        reload_on_cpu(&model_path)?.session()
    }
    Err(e) => Err(anyhow::anyhow!("Failed to create session: {e}")),
}

When it happens

Trigger: Calling load_model for a large GGUF with insufficient free RAM/VRAM for compute buffers; GPU context creation failing after load (driver reset, device lost); backend-specific session init quirks under Auto/CPU-fallback paths.

Common situations: Large-v3 models on 8 GB machines with other apps eating memory; GPU driver crash/reset between load and session creation; hybrid laptops switching GPUs mid-load; running alongside other local-LLM processes holding VRAM.

Related errors

• Device index ( ) cannot host a model
• Failed to load whisper model
• transcribe-cpp transcription failed
• get_available_accelerators panicked
• No compute device with index

AI-assisted analysis of cjpais/Handy@ba10ce1943 (2026-08-17). Data as JSON: /api/errors/5f6a5e53fa14e07e. Report an issue: GitHub.

Appendix: source

Thrown at src-tauri/src/managers/transcription.rs:606

                    .as_ref()
                    .map(transcribe_device_label)
                    .unwrap_or_else(|| "automatic".to_string());
                let model_options = ModelOptions { backend, device };
                let model = Model::load_with(&model_path, &model_options).map_err(|e| {
                    let error_msg = format!("Failed to load whisper model {}: {}", model_id, e);
                    emit_loading_failed(&error_msg);
                    anyhow::anyhow!(error_msg)
                })?;
                // The bound backend may differ from the request (e.g. CPU
                // fallback under Auto); log what actually loaded.
                let bound_backend = model.backend();
                let session = model.session().map_err(|e| {
                    let error_msg = format!(
                        "Failed to create session for whisper model {}: {}",
                        model_id, e
                    );
                    emit_loading_failed(&error_msg);
                    anyhow::anyhow!(error_msg)
                })?;
                // Reconcile the registry's advertised capabilities with the
                // loaded model's real ones (GGUF metadata) so badges/gating
                // reflect runtime truth, not the pre-download probe. The
                // load-completed event below triggers the frontend refresh.
                let caps = session.model().capabilities();
                self.model_manager.set_runtime_capabilities(
                    model_id,
                    caps.supports_streaming,
                    caps.supports_translate,
                    caps.supports_language_detect,
                    caps.languages.clone(),
                );
                let bound_device = model
                    .device()
                    .map(|device| transcribe_device_label(&device))
                    .unwrap_or_else(|_| "unknown".to_string());
                info!(

View on GitHub (pinned to ba10ce1943)