{"record":{"id":"b007865f2be2485b","repo":"rustfs/rustfs","slug":"gpu-device-error-0","errorCode":null,"errorMessage":"GPU device error: {0}","messagePattern":"GPU device error: (.+?)","errorType":"exception","errorClass":"GpuError","httpStatus":null,"severity":"error","filePath":"crates/obs/src/metrics/collectors/system_gpu.rs","lineNumber":72,"sourceCode":"///\n/// Contains GPU memory usage metrics for the monitored process.\n\n#[derive(Debug, Clone, Default)]\npub struct GpuStats {\n    /// GPU memory usage in bytes\n    pub memory_usage: u64,\n}\n\n/// GPU collector error types.\n\n#[derive(Debug, Error)]\npub enum GpuError {\n    /// GPU initialization failed\n    #[error(\"GPU initialization failed: {0}\")]\n    InitError(String),\n\n    /// GPU device access error\n    #[error(\"GPU device error: {0}\")]\n    DeviceError(String),\n\n    /// Process not found in GPU process list\n    #[error(\"Process not found in GPU process list\")]\n    ProcessNotFound,\n}\n\n/// GPU metrics collector.\n///\n/// Collects GPU memory usage metrics for a specific process using NVML.\npub struct GpuCollector {\n    /// NVML instance for GPU access\n    nvml: Nvml,\n    /// Process ID to monitor\n    pid: Pid,\n    /// Cached device count so we only probe NVML topology once at init time.\n    device_count: u32,\n}","sourceCodeStart":54,"sourceCodeEnd":90,"githubUrl":"https://github.com/rustfs/rustfs/blob/35af688cd9d41b4346fbe27dcf7250ba72046c1f/crates/obs/src/metrics/collectors/system_gpu.rs#L54-L90","documentation":"GpuError variant meaning a GPU device-level operation failed after initialization — NVML call error, device handle invalid, or device lost — while querying memory/process stats. The vendor/driver detail is embedded; the collector skips this sampling round and the operator should check driver health, since init succeeded but the device is now misbehaving.","triggerScenarios":"Thrown at crates/obs/src/metrics/collectors/system_gpu.rs:72 when the library encounters an invalid state.","commonSituations":"See trigger scenarios.","solutions":["Check GPU health with nvidia-smi on the host","Verify container/device permissions expose the GPU to the process","Restart the process or host if the driver is in a bad state"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"35af688cd9d41b4346fbe27dcf7250ba72046c1f","analyzedAt":"2026-08-20T21:57:04.799Z","contentChangedAt":"2026-08-20T21:57:04.799Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}