{"record":{"id":"c5db7e7972957fc7","repo":"henrygd/beszel","slug":"nvmlinit-failed-v","errorCode":null,"errorMessage":"nvmlInit failed: %v","messagePattern":"nvmlInit failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"agent/gpu_nvml.go","lineNumber":105,"sourceCode":"\tpurego.RegisterLibFunc(&nvmlShutdown, lib, \"nvmlShutdown\")\n\tpurego.RegisterLibFunc(&nvmlDeviceGetCount, lib, \"nvmlDeviceGetCount\")\n\tpurego.RegisterLibFunc(&nvmlDeviceGetHandleByIndex, lib, \"nvmlDeviceGetHandleByIndex\")\n\tpurego.RegisterLibFunc(&nvmlDeviceGetName, lib, \"nvmlDeviceGetName\")\n\t// Try to get v2 memory info, fallback to v1 if not available\n\tif hasSymbol(lib, \"nvmlDeviceGetMemoryInfo_v2\") {\n\t\tc.isV2 = true\n\t\tpurego.RegisterLibFunc(&nvmlDeviceGetMemoryInfo, lib, \"nvmlDeviceGetMemoryInfo_v2\")\n\t} else {\n\t\tpurego.RegisterLibFunc(&nvmlDeviceGetMemoryInfo, lib, \"nvmlDeviceGetMemoryInfo\")\n\t}\n\tpurego.RegisterLibFunc(&nvmlDeviceGetUtilizationRates, lib, \"nvmlDeviceGetUtilizationRates\")\n\tpurego.RegisterLibFunc(&nvmlDeviceGetTemperature, lib, \"nvmlDeviceGetTemperature\")\n\tpurego.RegisterLibFunc(&nvmlDeviceGetPowerUsage, lib, \"nvmlDeviceGetPowerUsage\")\n\tpurego.RegisterLibFunc(&nvmlDeviceGetPciInfo, lib, \"nvmlDeviceGetPciInfo\")\n\tpurego.RegisterLibFunc(&nvmlErrorString, lib, \"nvmlErrorString\")\n\n\tif ret := nvmlInit(); ret != nvmlReturn(nvmlSuccess) {\n\t\treturn fmt.Errorf(\"nvmlInit failed: %v\", ret)\n\t}\n\n\tvar count uint32\n\tif ret := nvmlDeviceGetCount(&count); ret != nvmlReturn(nvmlSuccess) {\n\t\treturn fmt.Errorf(\"nvmlDeviceGetCount failed: %v\", ret)\n\t}\n\n\tfor i := uint32(0); i < count; i++ {\n\t\tvar device nvmlDevice\n\t\tif ret := nvmlDeviceGetHandleByIndex(i, &device); ret == nvmlReturn(nvmlSuccess) {\n\t\t\tc.devices = append(c.devices, device)\n\t\t\t// Get BDF for power state check\n\t\t\tvar pci nvmlPciInfo\n\t\t\tif ret := nvmlDeviceGetPciInfo(device, &pci); ret == nvmlReturn(nvmlSuccess) {\n\t\t\t\tbusID := string(pci.BusId[:])\n\t\t\t\tif idx := strings.Index(busID, \"\\x00\"); idx != -1 {\n\t\t\t\t\tbusID = busID[:idx]\n\t\t\t\t}","sourceCodeStart":87,"sourceCodeEnd":123,"githubUrl":"https://github.com/henrygd/beszel/blob/b38fb7dafa60812cc22e6a84ce313e94f1ce0a32/agent/gpu_nvml.go#L87-L123","documentation":"The NVML library loaded successfully, but the nvmlInit() call returned a status other than nvmlSuccess. This means NVML itself rejected initialization — typically the driver is loaded but unusable (mismatched kernel/userland, no permission, driver in a bad state).","triggerScenarios":"nvmlCollector.init() in agent/gpu_nvml.go: purego-registered nvmlInit returns ret != nvmlSuccess; the error message embeds the raw NVML return code (e.g. 9 = NVML_ERROR_DRIVER_NOT_LOADED, 13 = NVML_ERROR_INSUFFICIENT_PERMISSIONS).","commonSituations":"nvidia kernel module loaded but userspace driver version mismatch after upgrade; driver installed inside container but kernel module absent on host; /dev/nvidia* device nodes not exposed; driver still initializing or crashed after resume from sleep.","solutions":["Decode the returned code with the NVML docs (e.g. 9 → driver not loaded) to target the fix.","Reinstall/align the NVIDIA driver so userspace libnvidia-ml matches the loaded kernel module.","Ensure /dev/nvidia*, /dev/nvidiactl are present and accessible in the environment.","Reboot or reload nvidia modules (rmmod/modprobe nvidia) after driver upgrades.","In containers, use --gpus all / NVIDIA runtime so the full driver stack is correctly wired."],"exampleFix":"// before: driver userspace updated without reloading kernel module\nsudo apt upgrade nvidia-driver-535\n// after\nsudo apt upgrade nvidia-driver-535 && sudo reboot","handlingStrategy":"retry","validationCode":"// verify driver userspace/kernel alignment before starting:\nnvidia-smi || dmesg | grep -i NVRM","typeGuard":null,"tryCatchPattern":"if err := c.init(); err != nil {\n    if strings.Contains(err.Error(), \"nvmlInit failed\") {\n        time.Sleep(5 * time.Second)\n        return c.init() // driver may still be initializing\n    }\n    return err\n}","preventionTips":["Reboot or reload nvidia modules after driver upgrades","Keep userspace libnvidia-ml and kernel module versions in lockstep","Expose /dev/nvidia* devices to containers via the NVIDIA runtime","Decode the embedded NVML return code to target the right fix (9=driver not loaded, 13=permissions)"],"tags":["gpu","nvidia","nvml","driver"],"backgroundTag":"nvml-init-failed","analyzedSha":"b38fb7dafa60812cc22e6a84ce313e94f1ce0a32","analyzedAt":"2026-08-31T15:10:10.149Z","schemaVersion":2},"datasetVersion":"2026-08-31T19:17:28.585Z"}