What causes "failed to stat %q: %w" in prometheus/node_exporter?

Make writers use atomic rename so files in the directory are never deleted while being scraped

failed to stat

Error message

failed to stat %q: %w

What it means

After a file parses successfully, processFile calls f.Stat() to obtain the modification time used as the mtime label for staleness detection. If Stat fails, the metrics cannot be safely timestamped, so the error is wrapped as this message (note it returns the parsed families alongside the error).

Solutions

Make writers use atomic rename so files in the directory are never deleted while being scraped
Point --collector.textfile.directory at a local filesystem (tmpfs/ext4), not NFS/FUSE
Re-run the scrape — this is usually transient; check whether node_exporter.textfile.scrape_error stays elevated
Check filesystem health (dmesg, mount options) if it persists on local disks

Defensive patterns

Strategy: retry

Validate before calling

if _, err := os.Stat(filepath.Join(dir, name)); err != nil {
    // file vanished; skip this scrape cycle
}

Try / catch

if err := coll.Update(ch); err != nil {
    if strings.Contains(err.Error(), "failed to stat") {
        // transient — retry on next scrape interval
        log.Debug("textfile stat failed transiently", "err", err)
    }
}

Prevention

Use atomic rename-based writes so files never vanish mid-scrape
Keep the textfile directory on a local filesystem, not NFS/FUSE
Alert on persistent node_exporter textfile scrape errors, not single occurrences
Periodically clean stale files from the textfile directory

When it happens

Trigger: f.Stat() fails on an already-open file — typically when the file was deleted/replaced on disk between open and stat (the fd is still valid but the stat call errors in unusual filesystem conditions), or I/O errors on exotic filesystems.

Common situations: Writers aggressively replacing/removing files in the textfile directory during scrape; NFS or FUSE mounts where stat on open handles can fail; permission changes mid-scrape.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

AI-assisted analysis of prometheus/node_exporter@17ddd77c59 (2026-09-07). Data as JSON: /api/errors/03dbd8c3190d67b9. Report an issue: GitHub.

Appendix: source

Thrown at collector/textfile.go:310

		return nil, nil, fmt.Errorf("failed to open textfile data file %q: %w", path, err)
	}
	defer f.Close()

	parser := expfmt.NewTextParser(model.UTF8Validation)
	families, err := parser.TextToMetricFamilies(f)
	if err != nil {
		return nil, nil, fmt.Errorf("failed to parse textfile data from %q: %w", path, err)
	}

	if hasTimestamps(families) {
		return nil, nil, fmt.Errorf("textfile %q contains unsupported client-side timestamps, skipping entire file", path)
	}

	// Only stat the file once it has been parsed and validated, so that
	// a failure does not appear fresh.
	stat, err := f.Stat()
	if err != nil {
		return nil, families, fmt.Errorf("failed to stat %q: %w", path, err)
	}

	t := stat.ModTime()
	return &t, families, nil
}

// hasTimestamps returns true when metrics contain unsupported timestamps.
func hasTimestamps(parsedFamilies map[string]*dto.MetricFamily) bool {
	for _, mf := range parsedFamilies {
		for _, m := range mf.Metric {
			if m.TimestampMs != nil {
				return true
			}
		}
	}
	return false
}

View on GitHub (pinned to 17ddd77c59)