henrygd/beszel · error

nvmlInit failed: %v

Error message

nvmlInit failed: %v

What it means

The NVML library loaded successfully, but the nvmlInit() call returned a status other than nvmlSuccess. This means NVML itself rejected initialization — typically the driver is loaded but unusable (mismatched kernel/userland, no permission, driver in a bad state).

Source

Thrown at agent/gpu_nvml.go:105

	purego.RegisterLibFunc(&nvmlShutdown, lib, "nvmlShutdown")
	purego.RegisterLibFunc(&nvmlDeviceGetCount, lib, "nvmlDeviceGetCount")
	purego.RegisterLibFunc(&nvmlDeviceGetHandleByIndex, lib, "nvmlDeviceGetHandleByIndex")
	purego.RegisterLibFunc(&nvmlDeviceGetName, lib, "nvmlDeviceGetName")
	// Try to get v2 memory info, fallback to v1 if not available
	if hasSymbol(lib, "nvmlDeviceGetMemoryInfo_v2") {
		c.isV2 = true
		purego.RegisterLibFunc(&nvmlDeviceGetMemoryInfo, lib, "nvmlDeviceGetMemoryInfo_v2")
	} else {
		purego.RegisterLibFunc(&nvmlDeviceGetMemoryInfo, lib, "nvmlDeviceGetMemoryInfo")
	}
	purego.RegisterLibFunc(&nvmlDeviceGetUtilizationRates, lib, "nvmlDeviceGetUtilizationRates")
	purego.RegisterLibFunc(&nvmlDeviceGetTemperature, lib, "nvmlDeviceGetTemperature")
	purego.RegisterLibFunc(&nvmlDeviceGetPowerUsage, lib, "nvmlDeviceGetPowerUsage")
	purego.RegisterLibFunc(&nvmlDeviceGetPciInfo, lib, "nvmlDeviceGetPciInfo")
	purego.RegisterLibFunc(&nvmlErrorString, lib, "nvmlErrorString")

	if ret := nvmlInit(); ret != nvmlReturn(nvmlSuccess) {
		return fmt.Errorf("nvmlInit failed: %v", ret)
	}

	var count uint32
	if ret := nvmlDeviceGetCount(&count); ret != nvmlReturn(nvmlSuccess) {
		return fmt.Errorf("nvmlDeviceGetCount failed: %v", ret)
	}

	for i := uint32(0); i < count; i++ {
		var device nvmlDevice
		if ret := nvmlDeviceGetHandleByIndex(i, &device); ret == nvmlReturn(nvmlSuccess) {
			c.devices = append(c.devices, device)
			// Get BDF for power state check
			var pci nvmlPciInfo
			if ret := nvmlDeviceGetPciInfo(device, &pci); ret == nvmlReturn(nvmlSuccess) {
				busID := string(pci.BusId[:])
				if idx := strings.Index(busID, "\x00"); idx != -1 {
					busID = busID[:idx]
				}

View on GitHub (pinned to b38fb7dafa)

Solutions

1. Decode the returned code with the NVML docs (e.g. 9 → driver not loaded) to target the fix.
2. Reinstall/align the NVIDIA driver so userspace libnvidia-ml matches the loaded kernel module.
3. Ensure /dev/nvidia*, /dev/nvidiactl are present and accessible in the environment.
4. Reboot or reload nvidia modules (rmmod/modprobe nvidia) after driver upgrades.
5. In containers, use --gpus all / NVIDIA runtime so the full driver stack is correctly wired.

Example fix

// before: driver userspace updated without reloading kernel module
sudo apt upgrade nvidia-driver-535
// after
sudo apt upgrade nvidia-driver-535 && sudo reboot

// verify driver userspace/kernel alignment before starting:
nvidia-smi || dmesg | grep -i NVRM

if err := c.init(); err != nil {
    if strings.Contains(err.Error(), "nvmlInit failed") {
        time.Sleep(5 * time.Second)
        return c.init() // driver may still be initializing
    }
    return err
}

When it happens

Trigger: nvmlCollector.init() in agent/gpu_nvml.go: purego-registered nvmlInit returns ret != nvmlSuccess; the error message embeds the raw NVML return code (e.g. 9 = NVML_ERROR_DRIVER_NOT_LOADED, 13 = NVML_ERROR_INSUFFICIENT_PERMISSIONS).

Common situations: nvidia kernel module loaded but userspace driver version mismatch after upgrade; driver installed inside container but kernel module absent on host; /dev/nvidia* device nodes not exposed; driver still initializing or crashed after resume from sleep.

Related errors

• nvmlDeviceGetCount failed: %v
• failed to load %s: %w
• nvml not supported on this platform
• no valid GPU data found
• scanner error: %w

AI-assisted analysis of henrygd/beszel@b38fb7dafa (2026-08-31). Data as JSON: /api/errors/c5db7e7972957fc7. Report an issue: GitHub.