{"record":{"id":"d1d6c55f13c939e1","repo":"henrygd/beszel","slug":"nvmldevicegetcount-failed-v","errorCode":null,"errorMessage":"nvmlDeviceGetCount failed: %v","messagePattern":"nvmlDeviceGetCount failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"agent/gpu_nvml.go","lineNumber":110,"sourceCode":"\tif hasSymbol(lib, \"nvmlDeviceGetMemoryInfo_v2\") {\n\t\tc.isV2 = true\n\t\tpurego.RegisterLibFunc(&nvmlDeviceGetMemoryInfo, lib, \"nvmlDeviceGetMemoryInfo_v2\")\n\t} else {\n\t\tpurego.RegisterLibFunc(&nvmlDeviceGetMemoryInfo, lib, \"nvmlDeviceGetMemoryInfo\")\n\t}\n\tpurego.RegisterLibFunc(&nvmlDeviceGetUtilizationRates, lib, \"nvmlDeviceGetUtilizationRates\")\n\tpurego.RegisterLibFunc(&nvmlDeviceGetTemperature, lib, \"nvmlDeviceGetTemperature\")\n\tpurego.RegisterLibFunc(&nvmlDeviceGetPowerUsage, lib, \"nvmlDeviceGetPowerUsage\")\n\tpurego.RegisterLibFunc(&nvmlDeviceGetPciInfo, lib, \"nvmlDeviceGetPciInfo\")\n\tpurego.RegisterLibFunc(&nvmlErrorString, lib, \"nvmlErrorString\")\n\n\tif ret := nvmlInit(); ret != nvmlReturn(nvmlSuccess) {\n\t\treturn fmt.Errorf(\"nvmlInit failed: %v\", ret)\n\t}\n\n\tvar count uint32\n\tif ret := nvmlDeviceGetCount(&count); ret != nvmlReturn(nvmlSuccess) {\n\t\treturn fmt.Errorf(\"nvmlDeviceGetCount failed: %v\", ret)\n\t}\n\n\tfor i := uint32(0); i < count; i++ {\n\t\tvar device nvmlDevice\n\t\tif ret := nvmlDeviceGetHandleByIndex(i, &device); ret == nvmlReturn(nvmlSuccess) {\n\t\t\tc.devices = append(c.devices, device)\n\t\t\t// Get BDF for power state check\n\t\t\tvar pci nvmlPciInfo\n\t\t\tif ret := nvmlDeviceGetPciInfo(device, &pci); ret == nvmlReturn(nvmlSuccess) {\n\t\t\t\tbusID := string(pci.BusId[:])\n\t\t\t\tif idx := strings.Index(busID, \"\\x00\"); idx != -1 {\n\t\t\t\t\tbusID = busID[:idx]\n\t\t\t\t}\n\t\t\t\tc.bdfs = append(c.bdfs, strings.ToLower(busID))\n\t\t\t} else {\n\t\t\t\tc.bdfs = append(c.bdfs, \"\")\n\t\t\t}\n\t\t}","sourceCodeStart":92,"sourceCodeEnd":128,"githubUrl":"https://github.com/henrygd/beszel/blob/b38fb7dafa60812cc22e6a84ce313e94f1ce0a32/agent/gpu_nvml.go#L92-L128","documentation":"nvmlInit succeeded but nvmlDeviceGetCount returned a non-success NVML status, so the collector cannot enumerate GPUs. The error embeds the NVML return code (e.g. 15 = NVML_ERROR_UNINITIALIZED, 13 = permissions).","triggerScenarios":"nvmlCollector.init() calls nvmlDeviceGetCount(&count) immediately after successful nvmlInit(); any non-nvmlSuccess return raises this error. Rare in practice since init just succeeded.","commonSituations":"Driver state changed between the two calls (module unloaded, GPU reset); restricted environments where NVML initializes but device enumeration is denied; flaky driver after suspend/resume; race with driver initialization at boot.","solutions":["Read the embedded NVML code (nvmlErrorString equivalent) to identify the exact status.","Restart the agent — transient failures right after nvmlInit usually clear on retry.","Verify the kernel module and devices remain healthy: nvidia-smi -L, check dmesg for NVRM errors.","Ensure GPU devices are passed through and writable in containers (--gpus all).","Reload the NVIDIA driver if the module entered a bad state (modprobe -r nvidia_uvm nvidia && modprobe nvidia)."],"exampleFix":"// before: agent starts at boot before driver is ready\nExecStart=/usr/local/bin/beszel-agent\n// after (systemd)\nExecStart=/usr/local/bin/beszel-agent\nAfter=nvidia-driver.service multi-user.target","handlingStrategy":"retry","validationCode":"nvidia-smi -L  # enumerating devices should succeed before agent start","typeGuard":null,"tryCatchPattern":"if ret := nvmlDeviceGetCount(&count); ret != nvmlSuccess {\n    return retry.WithDelay(func() error { return c.init() }, 3, 5*time.Second)\n}","preventionTips":["Start the agent after the NVIDIA driver service (systemd After=)","Monitor dmesg NVRM errors and reload modules on bad states","Retry collector init with backoff on transient NVML codes","Ensure GPUs are not mid-reset (GPU reset / suspend-resume) when the agent starts"],"tags":["gpu","nvidia","nvml","driver"],"backgroundTag":"nvml-init-failed","analyzedSha":"b38fb7dafa60812cc22e6a84ce313e94f1ce0a32","analyzedAt":"2026-08-31T15:10:10.149Z","schemaVersion":2},"datasetVersion":"2026-08-31T19:17:28.585Z"}