{"record":{"id":"30595a250e1b95c9","repo":"prometheus/node_exporter","slug":"error-reading-stat-for-pid-d-w","errorCode":null,"errorMessage":"error reading stat for pid %d: %w","messagePattern":"error reading stat for pid (.+?): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"collector/processes_linux.go","lineNumber":135,"sourceCode":"\tp, err := c.fs.AllProcs()\n\tif err != nil {\n\t\treturn 0, nil, 0, nil, fmt.Errorf(\"unable to list all processes: %w\", err)\n\t}\n\tpids := 0\n\tthread := 0\n\tprocStates := make(map[string]int32)\n\tthreadStates := make(map[string]int32)\n\n\tfor _, pid := range p {\n\t\tstat, err := pid.Stat()\n\t\tif err != nil {\n\t\t\t// PIDs can vanish between getting the list and getting stats.\n\t\t\tif c.isIgnoredError(err) {\n\t\t\t\tc.logger.Debug(\"file not found when retrieving stats for pid\", \"pid\", pid.PID, \"err\", err)\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tc.logger.Debug(\"error reading stat for pid\", \"pid\", pid.PID, \"err\", err)\n\t\t\treturn 0, nil, 0, nil, fmt.Errorf(\"error reading stat for pid %d: %w\", pid.PID, err)\n\t\t}\n\t\tpids++\n\t\tprocStates[stat.State]++\n\t\tthread += stat.NumThreads\n\t\terr = c.getThreadStates(pid.PID, stat, threadStates)\n\t\tif err != nil {\n\t\t\treturn 0, nil, 0, nil, err\n\t\t}\n\t}\n\treturn pids, procStates, thread, threadStates, nil\n}\n\nfunc (c *processCollector) getThreadStates(pid int, pidStat procfs.ProcStat, threadStates map[string]int32) error {\n\tfs, err := procfs.NewFS(procFilePath(path.Join(strconv.Itoa(pid), \"task\")))\n\tif err != nil {\n\t\tif c.isIgnoredError(err) {\n\t\t\tc.logger.Debug(\"file not found when retrieving tasks for pid\", \"pid\", pid, \"err\", err)\n\t\t\treturn nil","sourceCodeStart":117,"sourceCodeEnd":153,"githubUrl":"https://github.com/prometheus/node_exporter/blob/17ddd77c59ba27e1508e9f7894b1e55b44d6aed3/collector/processes_linux.go#L117-L153","documentation":"While iterating processes, getAllocatedThreads reads each pid's stat via procfs; errors that are not 'ignored' (ENOENT-class races) are wrapped as 'error reading stat for pid %d'. PIDs can vanish between listing and stat reading, which is why ignored errors are skipped with a debug log, but any other failure aborts the whole scrape update.","triggerScenarios":"pid.Stat() (procfs ProcStat read of /proc/<pid>/stat) fails for some pid and c.isIgnoredError(err) is false, so the function returns the wrapped error immediately.","commonSituations":"Permission errors on /proc/<pid>/stat for processes owned by other users (hidepid=2); kernel bugs or corrupted proc entries; reading via a stale --path.procfs pointing at a snapshot where stat parsing fails.","solutions":["Read the wrapped error: EACCES/EPERM means fix permissions (run exporter as root, or adjust hidepid mount options); ENOENT should normally be ignored already.","If hidepid=2 is set on the procfs mount, remount with hidepid=gid=<exporter-group> so the exporter can read all processes' stat files.","Update procfs/node_exporter: newer versions ignore a broader set of transient errors.","Check the debug log line 'error reading stat for pid' for the specific pid and reproduce with cat /proc/<pid>/stat."],"exampleFix":"// before\nmount -o remount,hidepid=2 /proc\n// after\nmount -o remount,hidepid=2,gid=$(id -g node_exporter) /proc","handlingStrategy":"retry","validationCode":"// sanity check: exporter user can read an arbitrary proc stat\nf, err := os.Open(\"/proc/1/stat\")\nif err != nil {\n    log.Printf(\"cannot read other processes' stat (hidepid?): %v\", err)\n} else {\n    f.Close()\n}","typeGuard":null,"tryCatchPattern":"if err := c.Update(ch); err != nil {\n    var pidErr pidStatError // inspect wrapped cause\n    if errors.Is(err, os.ErrPermission) {\n        log.Printf(\"fix hidepid/permissions for the exporter user: %v\", err)\n    } else {\n        // likely a pid race; retry on next scrape\n    }\n}","preventionTips":["Remount procfs with hidepid=gid=<exporter-group> instead of hidepid=2.","Run node_exporter as root in environments with hostile proc permissions.","Upgrade procfs so vanished-pid errors are classified as ignorable.","Distinguish single-pid failures (races/permissions) from total scrape failures."],"tags":["linux","procfs","processes","hidepid","permissions","node-exporter"],"backgroundTag":"file-read-failed","analyzedSha":"17ddd77c59ba27e1508e9f7894b1e55b44d6aed3","analyzedAt":"2026-09-07T17:54:06.211Z","contentChangedAt":"2026-09-07T17:54:06.211Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}