prometheus/node_exporter · error

unable to retrieve number of allocated threads

Error message

unable to retrieve number of allocated threads: %w

What it means

processCollector.Update wraps errors from getAllocatedThreads with this message. getAllocatedThreads enumerates all processes via procfs (fs.AllProcs()) and reads each /proc/<pid>/stat; any failure listing processes or reading stats surfaces here as 'unable to retrieve number of allocated threads'.

Solutions

1. Check the wrapped cause (%w) in logs: if it's 'unable to list all processes' or 'error reading stat for pid', inspect /proc access for the exporter user.
2. If hidepid is in effect, run the exporter as root or grant it the required group (hidepid=gid=...) so it can read all /proc/<pid>/stat entries.
3. Confirm the failing read is transient (pid race) — node_exporter already ignores ENOENT-style errors via isIgnoredError; persistent failures indicate a permissions or procfs issue.
4. Re-run the scrape; a single failed scrape with this error during process churn is usually harmless.

// before scraping
if _, err := os.Stat("/proc/self/stat"); err != nil {
    log.Printf("procfs unreadable for exporter user: %v", err)
}

if err := c.Update(ch); err != nil {
    var transient bool
    if strings.Contains(err.Error(), "no such file") {
        transient = true // pid race: safe to retry next scrape
    }
    log.Printf("processes update failed (transient=%v): %v", transient, err)
}

When it happens

Trigger: Update() is called on a scrape and c.getAllocatedThreads() returns an error: fs.AllProcs() failed (145), a pid's stat read failed non-fatally (146), or getThreadStates failed (147-149).

Common situations: Scraping on a system where /proc/<pid> entries vanish rapidly (short-lived processes under heavy churn); restricted permissions on /proc; hidepid=2 mount option hiding other users' processes from the exporter user.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• unable to list all processes
• error reading stat for pid
• couldn't get buddyinfo
• failed to get memory info
• couldn't get netstats

AI-assisted analysis of prometheus/node_exporter@17ddd77c59 (2026-09-07). Data as JSON: /api/errors/706d7cc52fb9ee86. Report an issue: GitHub.

Appendix: source

Thrown at collector/processes_linux.go:88

		),
		procsState: prometheus.NewDesc(
			prometheus.BuildFQName(namespace, subsystem, "state"),
			"Number of processes in each state.",
			[]string{"state"}, nil,
		),
		pidUsed: prometheus.NewDesc(prometheus.BuildFQName(namespace, subsystem, "pids"),
			"Number of PIDs", nil, nil,
		),
		pidMax: prometheus.NewDesc(prometheus.BuildFQName(namespace, subsystem, "max_processes"),
			"Number of max PIDs limit", nil, nil,
		),
		logger: logger,
	}, nil
}
func (c *processCollector) Update(ch chan<- prometheus.Metric) error {
	pids, states, threads, threadStates, err := c.getAllocatedThreads()
	if err != nil {
		return fmt.Errorf("unable to retrieve number of allocated threads: %w", err)
	}

	ch <- prometheus.MustNewConstMetric(c.threadAlloc, prometheus.GaugeValue, float64(threads))
	maxThreads, err := readUintFromFile(procFilePath("sys/kernel/threads-max"))
	if err != nil {
		return fmt.Errorf("unable to retrieve limit number of threads: %w", err)
	}
	ch <- prometheus.MustNewConstMetric(c.threadLimit, prometheus.GaugeValue, float64(maxThreads))

	for state := range states {
		ch <- prometheus.MustNewConstMetric(c.procsState, prometheus.GaugeValue, float64(states[state]), state)
	}

	for state := range threadStates {
		ch <- prometheus.MustNewConstMetric(c.threadsState, prometheus.GaugeValue, float64(threadStates[state]), state)
	}

	pidM, err := readUintFromFile(procFilePath("sys/kernel/pid_max"))

View on GitHub (pinned to 17ddd77c59)