prometheus/node_exporter · error

error reading stat for pid

Error message

error reading stat for pid:%d thread:%d err:%w

What it means

For each thread in /proc/<pid>/task, getThreadStates reads the thread's stat file; non-ignored errors are wrapped as 'error reading stat for pid:%d thread:%d err:%w'. Vanished threads (ENOENT-style) are skipped via isIgnoredError, so this error indicates a genuine read/parse/permission failure on a thread's stat.

Solutions

1. Inspect the wrapped cause from the debug log ('error reading stat for thread') and test manually: cat /proc/<pid>/task/<tid>/stat as the exporter user.
2. Fix permissions: run node_exporter as root or configure hidepid=gid=<exporter-group> / LSM rules to allow reading all processes' stat files.
3. Upgrade procfs/node_exporter so transient disappearances are ignored as ENOENT-class errors.
4. If only a snapshot fixture is being read (tests), regenerate fixtures with make update_fixtures so stat files parse correctly.

// exporter user must read thread stat files
b, err := os.ReadFile("/proc/self/task/1/stat")
if err != nil {
    log.Printf("cannot read thread stat: %v", err)
} else {
    _ = b
}

if err := c.Update(ch); err != nil {
    if errors.Is(err, os.ErrPermission) {
        log.Printf("thread stat permission denied; adjust hidepid/LSM policy: %v", err)
    } else if errors.Is(err, os.ErrNotExist) {
        // thread exited mid-scrape: safe to ignore/retry
    }
}

When it happens

Trigger: thread.Stat() fails for a thread TID inside /proc/<pid>/task and isIgnoredError(err) is false, causing getThreadStates — and therefore the whole processes Update — to return an error.

Common situations: Permission denied on other users' thread stat files under hidepid=2; SELinux denials on procfs; a stale fixture-based procfs whose stat files don't parse; very high thread churn where a thread disappears with an unexpected error type.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• error reading stat for pid
• error reading task for pid
• unable to list all threads for pid
• unable to retrieve limit number of threads
• failed to open procfs

AI-assisted analysis of prometheus/node_exporter@17ddd77c59 (2026-09-07). Data as JSON: /api/errors/5452ba28c6a972e1. Report an issue: GitHub.

Appendix: source

Thrown at collector/processes_linux.go:180

			c.logger.Debug("file not found when retrieving tasks for pid", "pid", pid, "err", err)
			return nil
		}
		return fmt.Errorf("unable to list all threads for pid: %d %w", pid, err)
	}

	for _, thread := range t {
		if pid == thread.PID {
			threadStates[pidStat.State]++
			continue
		}
		threadStat, err := thread.Stat()
		if err != nil {
			if c.isIgnoredError(err) {
				c.logger.Debug("file not found when retrieving stats for thread", "pid", pid, "threadId", thread.PID, "err", err)
				continue
			}
			c.logger.Debug("error reading stat for thread", "pid", pid, "threadId", thread.PID, "err", err)
			return fmt.Errorf("error reading stat for pid:%d thread:%d err:%w", pid, thread.PID, err)
		}
		threadStates[threadStat.State]++
	}
	return nil
}

func (c *processCollector) isIgnoredError(err error) bool {
	if errors.Is(err, os.ErrNotExist) || strings.Contains(err.Error(), syscall.ESRCH.Error()) {
		return true
	}
	return false
}

View on GitHub (pinned to 17ddd77c59)