rustfs/rustfs · error · GpuError

GPU device error: {0}

Error message

GPU device error: {0}

What it means

GpuError variant meaning a GPU device-level operation failed after initialization — NVML call error, device handle invalid, or device lost — while querying memory/process stats. The vendor/driver detail is embedded; the collector skips this sampling round and the operator should check driver health, since init succeeded but the device is now misbehaving.

Source

Thrown at crates/obs/src/metrics/collectors/system_gpu.rs:72

///
/// Contains GPU memory usage metrics for the monitored process.

#[derive(Debug, Clone, Default)]
pub struct GpuStats {
    /// GPU memory usage in bytes
    pub memory_usage: u64,
}

/// GPU collector error types.

#[derive(Debug, Error)]
pub enum GpuError {
    /// GPU initialization failed
    #[error("GPU initialization failed: {0}")]
    InitError(String),

    /// GPU device access error
    #[error("GPU device error: {0}")]
    DeviceError(String),

    /// Process not found in GPU process list
    #[error("Process not found in GPU process list")]
    ProcessNotFound,
}

/// GPU metrics collector.
///
/// Collects GPU memory usage metrics for a specific process using NVML.
pub struct GpuCollector {
    /// NVML instance for GPU access
    nvml: Nvml,
    /// Process ID to monitor
    pid: Pid,
    /// Cached device count so we only probe NVML topology once at init time.
    device_count: u32,
}

View on GitHub (pinned to 35af688cd9)

Solutions

1. Check GPU health with nvidia-smi on the host
2. Verify container/device permissions expose the GPU to the process
3. Restart the process or host if the driver is in a bad state

When it happens

Trigger: Thrown at crates/obs/src/metrics/collectors/system_gpu.rs:72 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rustfs/rustfs@35af688cd9 (2026-08-20). Data as JSON: /api/errors/b007865f2be2485b. Report an issue: GitHub.