prometheus/node_exporter · error

failed to scan NVMe subsystems

Error message

failed to scan NVMe subsystems: %w

What it means

Update reads the NVMe subsystem class directory through c.fs.NVMeSubsystemClass(). If the scan fails with an error other than os.ErrNotExist/os.ErrPermission (which are downgraded to ErrNoData), it is wrapped as "failed to scan NVMe subsystems" and propagated as a scrape error. It represents an I/O or traversal problem while enumerating /sys/class/nvme-subsystem/.

Solutions

1. Check the wrapped cause (%w) in the log to identify the underlying syscall error
2. Re-scrape; transient hotplug-related races typically resolve on the next scrape
3. Fix the failing sysfs hardware/medium if errors are persistent (EIO)
4. Verify the fixture/test root contains a well-formed class/nvme-subsystem directory
5. If the path legitimately may not exist, rely on ErrNoData handling already present for ErrNotExist

if fi, err := os.Stat(filepath.Join(*sysPath, "class", "nvme-subsystem")); err != nil || !fi.IsDir() { log.Println("nvme-subsystem class dir unavailable; ErrNoData expected") }

func nvmeClassReadable(sysPath string) bool {
    f, err := os.Open(filepath.Join(sysPath, "class", "nvme-subsystem"))
    if err != nil { return false }
    f.Close()
    return true
}

if err := c.Update(ch); err != nil {
    var nd collector.ErrNoData
    if errors.As(err, &nd) { return nil }
    if strings.Contains(err.Error(), "failed to scan NVMe subsystems") {
        logger.Warn("transient NVMe scan failure, will retry next scrape", "err", err)
        return nil
    }
    return err
}

When it happens

Trigger: c.fs.NVMeSubsystemClass() returns an error that is neither ErrNotExist nor ErrPermission: e.g. the sysfs FS was constructed against a fixture/test root where reading class/nvme-subsystem fails with another I/O error, a readlink/readFile error while parsing subsystem attributes, or a corrupt/inaccessible entry.

Common situations: Scraping while sysfs entries disappear mid-scan (device hot-unplug); restricted environments where read errors are neither ENOENT nor EACCES (e.g. EIO from failing hardware); tests pointing the collector at a partially-populated fixture tree.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• failed to retrieve bcache stats
• failed to retrieve bcachefs stats
• failed to retrieve Btrfs stats from procfs
• failed to scan DM-multipath devices
• couldn't get NUMA meminfo

AI-assisted analysis of prometheus/node_exporter@17ddd77c59 (2026-09-07). Data as JSON: /api/errors/9b1f0e3b21bb84fa. Report an issue: GitHub.

Appendix: source

Thrown at collector/nvmesubsystem_linux.go:121

	fs, err := sysfs.NewFS(*sysPath)
	if err != nil {
		return nil, fmt.Errorf("failed to open sysfs: %w", err)
	}

	return &nvmeSubsystemCollector{
		fs:     fs,
		logger: logger,
	}, nil
}

func (c *nvmeSubsystemCollector) Update(ch chan<- prometheus.Metric) error {
	subsystems, err := c.fs.NVMeSubsystemClass()
	if err != nil {
		if errors.Is(err, os.ErrNotExist) || errors.Is(err, os.ErrPermission) {
			c.logger.Debug("Could not read NVMe subsystem info", "err", err)
			return ErrNoData
		}
		return fmt.Errorf("failed to scan NVMe subsystems: %w", err)
	}

	for _, subsys := range subsystems {
		ch <- prometheus.MustNewConstMetric(nvmesubsystemInfo, prometheus.GaugeValue, 1,
			subsys.Name, subsys.NQN, subsys.Model, subsys.Serial, subsys.IOPolicy)

		for _, ns := range subsys.Namespaces {
			ch <- prometheus.MustNewConstMetric(nvmesubsystemNamespaceInfo, prometheus.GaugeValue, 1,
				subsys.Name, ns)
		}

		total := float64(len(subsys.Controllers))
		var live float64
		for _, ctrl := range subsys.Controllers {
			state := normalizeControllerState(ctrl.State)
			if state == "live" {
				live++
			}

View on GitHub (pinned to 17ddd77c59)