{"record":{"id":"5452ba28c6a972e1","repo":"prometheus/node_exporter","slug":"error-reading-stat-for-pid-d-thread-d-err-w","errorCode":null,"errorMessage":"error reading stat for pid:%d thread:%d err:%w","messagePattern":"error reading stat for pid:(.+?) thread:(.+?) err:%w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"collector/processes_linux.go","lineNumber":180,"sourceCode":"\t\t\tc.logger.Debug(\"file not found when retrieving tasks for pid\", \"pid\", pid, \"err\", err)\n\t\t\treturn nil\n\t\t}\n\t\treturn fmt.Errorf(\"unable to list all threads for pid: %d %w\", pid, err)\n\t}\n\n\tfor _, thread := range t {\n\t\tif pid == thread.PID {\n\t\t\tthreadStates[pidStat.State]++\n\t\t\tcontinue\n\t\t}\n\t\tthreadStat, err := thread.Stat()\n\t\tif err != nil {\n\t\t\tif c.isIgnoredError(err) {\n\t\t\t\tc.logger.Debug(\"file not found when retrieving stats for thread\", \"pid\", pid, \"threadId\", thread.PID, \"err\", err)\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tc.logger.Debug(\"error reading stat for thread\", \"pid\", pid, \"threadId\", thread.PID, \"err\", err)\n\t\t\treturn fmt.Errorf(\"error reading stat for pid:%d thread:%d err:%w\", pid, thread.PID, err)\n\t\t}\n\t\tthreadStates[threadStat.State]++\n\t}\n\treturn nil\n}\n\nfunc (c *processCollector) isIgnoredError(err error) bool {\n\tif errors.Is(err, os.ErrNotExist) || strings.Contains(err.Error(), syscall.ESRCH.Error()) {\n\t\treturn true\n\t}\n\treturn false\n}\n","sourceCodeStart":162,"sourceCodeEnd":193,"githubUrl":"https://github.com/prometheus/node_exporter/blob/17ddd77c59ba27e1508e9f7894b1e55b44d6aed3/collector/processes_linux.go#L162-L193","documentation":"For each thread in /proc/<pid>/task, getThreadStates reads the thread's stat file; non-ignored errors are wrapped as 'error reading stat for pid:%d thread:%d err:%w'. Vanished threads (ENOENT-style) are skipped via isIgnoredError, so this error indicates a genuine read/parse/permission failure on a thread's stat.","triggerScenarios":"thread.Stat() fails for a thread TID inside /proc/<pid>/task and isIgnoredError(err) is false, causing getThreadStates — and therefore the whole processes Update — to return an error.","commonSituations":"Permission denied on other users' thread stat files under hidepid=2; SELinux denials on procfs; a stale fixture-based procfs whose stat files don't parse; very high thread churn where a thread disappears with an unexpected error type.","solutions":["Inspect the wrapped cause from the debug log ('error reading stat for thread') and test manually: cat /proc/<pid>/task/<tid>/stat as the exporter user.","Fix permissions: run node_exporter as root or configure hidepid=gid=<exporter-group> / LSM rules to allow reading all processes' stat files.","Upgrade procfs/node_exporter so transient disappearances are ignored as ENOENT-class errors.","If only a snapshot fixture is being read (tests), regenerate fixtures with make update_fixtures so stat files parse correctly."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// exporter user must read thread stat files\nb, err := os.ReadFile(\"/proc/self/task/1/stat\")\nif err != nil {\n    log.Printf(\"cannot read thread stat: %v\", err)\n} else {\n    _ = b\n}","typeGuard":null,"tryCatchPattern":"if err := c.Update(ch); err != nil {\n    if errors.Is(err, os.ErrPermission) {\n        log.Printf(\"thread stat permission denied; adjust hidepid/LSM policy: %v\", err)\n    } else if errors.Is(err, os.ErrNotExist) {\n        // thread exited mid-scrape: safe to ignore/retry\n    }\n}","preventionTips":["Verify readability of /proc/<pid>/task/<tid>/stat for the exporter user.","Prefer hidepid=gid=<exporter-group> over hidepid=2.","Grant SELinux/AppArmor procfs read access to node_exporter.","Regenerate test fixtures with make update_fixtures if stat parsing fails in tests."],"tags":["linux","procfs","threads","permissions","hidepid","node-exporter"],"backgroundTag":"file-read-failed","analyzedSha":"17ddd77c59ba27e1508e9f7894b1e55b44d6aed3","analyzedAt":"2026-09-07T17:54:06.211Z","contentChangedAt":"2026-09-07T17:54:06.211Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}