prometheus/node_exporter · error

error reading stat for pid

Error message

error reading stat for pid %d: %w

What it means

While iterating processes, getAllocatedThreads reads each pid's stat via procfs; errors that are not 'ignored' (ENOENT-class races) are wrapped as 'error reading stat for pid %d'. PIDs can vanish between listing and stat reading, which is why ignored errors are skipped with a debug log, but any other failure aborts the whole scrape update.

Solutions

1. Read the wrapped error: EACCES/EPERM means fix permissions (run exporter as root, or adjust hidepid mount options); ENOENT should normally be ignored already.
2. If hidepid=2 is set on the procfs mount, remount with hidepid=gid=<exporter-group> so the exporter can read all processes' stat files.
3. Update procfs/node_exporter: newer versions ignore a broader set of transient errors.
4. Check the debug log line 'error reading stat for pid' for the specific pid and reproduce with cat /proc/<pid>/stat.

Example fix

// before
mount -o remount,hidepid=2 /proc
// after
mount -o remount,hidepid=2,gid=$(id -g node_exporter) /proc

// sanity check: exporter user can read an arbitrary proc stat
f, err := os.Open("/proc/1/stat")
if err != nil {
    log.Printf("cannot read other processes' stat (hidepid?): %v", err)
} else {
    f.Close()
}

if err := c.Update(ch); err != nil {
    var pidErr pidStatError // inspect wrapped cause
    if errors.Is(err, os.ErrPermission) {
        log.Printf("fix hidepid/permissions for the exporter user: %v", err)
    } else {
        // likely a pid race; retry on next scrape
    }
}

When it happens

Trigger: pid.Stat() (procfs ProcStat read of /proc/<pid>/stat) fails for some pid and c.isIgnoredError(err) is false, so the function returns the wrapped error immediately.

Common situations: Permission errors on /proc/<pid>/stat for processes owned by other users (hidepid=2); kernel bugs or corrupted proc entries; reading via a stale --path.procfs pointing at a snapshot where stat parsing fails.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• error reading stat for pid
• unable to retrieve number of allocated threads
• unable to list all processes
• error reading task for pid
• unable to list all threads for pid

AI-assisted analysis of prometheus/node_exporter@17ddd77c59 (2026-09-07). Data as JSON: /api/errors/30595a250e1b95c9. Report an issue: GitHub.

Appendix: source

Thrown at collector/processes_linux.go:135

	p, err := c.fs.AllProcs()
	if err != nil {
		return 0, nil, 0, nil, fmt.Errorf("unable to list all processes: %w", err)
	}
	pids := 0
	thread := 0
	procStates := make(map[string]int32)
	threadStates := make(map[string]int32)

	for _, pid := range p {
		stat, err := pid.Stat()
		if err != nil {
			// PIDs can vanish between getting the list and getting stats.
			if c.isIgnoredError(err) {
				c.logger.Debug("file not found when retrieving stats for pid", "pid", pid.PID, "err", err)
				continue
			}
			c.logger.Debug("error reading stat for pid", "pid", pid.PID, "err", err)
			return 0, nil, 0, nil, fmt.Errorf("error reading stat for pid %d: %w", pid.PID, err)
		}
		pids++
		procStates[stat.State]++
		thread += stat.NumThreads
		err = c.getThreadStates(pid.PID, stat, threadStates)
		if err != nil {
			return 0, nil, 0, nil, err
		}
	}
	return pids, procStates, thread, threadStates, nil
}

func (c *processCollector) getThreadStates(pid int, pidStat procfs.ProcStat, threadStates map[string]int32) error {
	fs, err := procfs.NewFS(procFilePath(path.Join(strconv.Itoa(pid), "task")))
	if err != nil {
		if c.isIgnoredError(err) {
			c.logger.Debug("file not found when retrieving tasks for pid", "pid", pid, "err", err)
			return nil

View on GitHub (pinned to 17ddd77c59)