henrygd/beszel · error

nvmlDeviceGetCount failed: %v

Error message

nvmlDeviceGetCount failed: %v

What it means

nvmlInit succeeded but nvmlDeviceGetCount returned a non-success NVML status, so the collector cannot enumerate GPUs. The error embeds the NVML return code (e.g. 15 = NVML_ERROR_UNINITIALIZED, 13 = permissions).

Source

Thrown at agent/gpu_nvml.go:110

	if hasSymbol(lib, "nvmlDeviceGetMemoryInfo_v2") {
		c.isV2 = true
		purego.RegisterLibFunc(&nvmlDeviceGetMemoryInfo, lib, "nvmlDeviceGetMemoryInfo_v2")
	} else {
		purego.RegisterLibFunc(&nvmlDeviceGetMemoryInfo, lib, "nvmlDeviceGetMemoryInfo")
	}
	purego.RegisterLibFunc(&nvmlDeviceGetUtilizationRates, lib, "nvmlDeviceGetUtilizationRates")
	purego.RegisterLibFunc(&nvmlDeviceGetTemperature, lib, "nvmlDeviceGetTemperature")
	purego.RegisterLibFunc(&nvmlDeviceGetPowerUsage, lib, "nvmlDeviceGetPowerUsage")
	purego.RegisterLibFunc(&nvmlDeviceGetPciInfo, lib, "nvmlDeviceGetPciInfo")
	purego.RegisterLibFunc(&nvmlErrorString, lib, "nvmlErrorString")

	if ret := nvmlInit(); ret != nvmlReturn(nvmlSuccess) {
		return fmt.Errorf("nvmlInit failed: %v", ret)
	}

	var count uint32
	if ret := nvmlDeviceGetCount(&count); ret != nvmlReturn(nvmlSuccess) {
		return fmt.Errorf("nvmlDeviceGetCount failed: %v", ret)
	}

	for i := uint32(0); i < count; i++ {
		var device nvmlDevice
		if ret := nvmlDeviceGetHandleByIndex(i, &device); ret == nvmlReturn(nvmlSuccess) {
			c.devices = append(c.devices, device)
			// Get BDF for power state check
			var pci nvmlPciInfo
			if ret := nvmlDeviceGetPciInfo(device, &pci); ret == nvmlReturn(nvmlSuccess) {
				busID := string(pci.BusId[:])
				if idx := strings.Index(busID, "\x00"); idx != -1 {
					busID = busID[:idx]
				}
				c.bdfs = append(c.bdfs, strings.ToLower(busID))
			} else {
				c.bdfs = append(c.bdfs, "")
			}
		}

View on GitHub (pinned to b38fb7dafa)

Solutions

1. Read the embedded NVML code (nvmlErrorString equivalent) to identify the exact status.
2. Restart the agent — transient failures right after nvmlInit usually clear on retry.
3. Verify the kernel module and devices remain healthy: nvidia-smi -L, check dmesg for NVRM errors.
4. Ensure GPU devices are passed through and writable in containers (--gpus all).
5. Reload the NVIDIA driver if the module entered a bad state (modprobe -r nvidia_uvm nvidia && modprobe nvidia).

Example fix

// before: agent starts at boot before driver is ready
ExecStart=/usr/local/bin/beszel-agent
// after (systemd)
ExecStart=/usr/local/bin/beszel-agent
After=nvidia-driver.service multi-user.target

nvidia-smi -L  # enumerating devices should succeed before agent start

if ret := nvmlDeviceGetCount(&count); ret != nvmlSuccess {
    return retry.WithDelay(func() error { return c.init() }, 3, 5*time.Second)
}

When it happens

Trigger: nvmlCollector.init() calls nvmlDeviceGetCount(&count) immediately after successful nvmlInit(); any non-nvmlSuccess return raises this error. Rare in practice since init just succeeded.

Common situations: Driver state changed between the two calls (module unloaded, GPU reset); restricted environments where NVML initializes but device enumeration is denied; flaky driver after suspend/resume; race with driver initialization at boot.

Related errors

• nvmlInit failed: %v
• failed to load %s: %w
• nvml not supported on this platform
• no valid GPU data found
• scanner error: %w

AI-assisted analysis of henrygd/beszel@b38fb7dafa (2026-08-31). Data as JSON: /api/errors/d1d6c55f13c939e1. Report an issue: GitHub.