What causes "line scan did not return 2 fields: %s" in prometheus/node_exporter?

Inspect /sys/devices/system/node/node*/numastat for the line with unexpected column count

line scan did not return 2 fields

Error message

line scan did not return 2 fields: %s

What it means

This error is returned by parseMemInfoNumaStat in collector/meminfo_numa_linux.go:172 when a non-empty line from /sys/devices/system/node/nodeN/numastat splits into a number of whitespace-separated fields other than exactly 2. The numastat format is 'stat_name value'; any deviation (extra columns, merged tokens) aborts parsing for that node.

Solutions

Inspect /sys/devices/system/node/node*/numastat for the line with unexpected column count
Verify kernel version; if numastat gained columns, update the parser to select the expected fields
Re-read the file to rule out truncated reads racing with kernel updates
Rebuild with -tags nomeminfo_numa to disable this collector on affected platforms
Report the nonstandard numastat format upstream if a stock kernel produces it

Example fix

// before
parts := strings.Fields(line)
if len(parts) != 2 {
	return nil, fmt.Errorf("line scan did not return 2 fields: %s", line)
}
// after (tolerate extra columns by taking first two)
parts := strings.Fields(line)
if len(parts) < 2 {
	return nil, fmt.Errorf("line scan did not return 2 fields: %s", line)
}

Defensive patterns

Strategy: validation

Validate before calling

// Validate numastat line has exactly 2 fields before parsing
parts := strings.Fields(line)
if len(parts) != 2 {
	return fmt.Errorf("expected 2 fields, got %d in %q", len(parts), line)
}

Try / catch

_, err := parseMemInfoNumaStat(f, node)
if err != nil {
	if strings.Contains(err.Error(), "line scan did not return 2 fields") {
		// degrade gracefully: skip numastat metrics, keep base meminfo
	}
	return err
}

Prevention

Verify `cat /sys/devices/system-node/*/numastat` format after kernel upgrades
Run e2e tests (make test-e2e) on target platforms before rollout
Avoid patched kernels that alter numastat layout without updating the parser
Alert on node_exporter scrape failures to detect format regressions

When it happens

Trigger: parseMemInfoNumaStat reads a non-empty line whose strings.Fields result has length != 2, e.g. 'hit 100 extra' or 'hit' (single field), from a node numastat file.

Common situations: Custom or patched kernels adding columns to numastat; virtualized environments exposing nonstandard numastat files; corrupted or fixture files used in tests; locale/whitespace oddities is unlikely but merged tokens can occur.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

AI-assisted analysis of prometheus/node_exporter@17ddd77c59 (2026-09-07). Data as JSON: /api/errors/a39e6e2335509777. Report an issue: GitHub.

Appendix: source

Thrown at collector/meminfo_numa_linux.go:172

	}

	return memInfo, scanner.Err()
}

func parseMemInfoNumaStat(r io.Reader, nodeNumber string) ([]meminfoMetric, error) {
	var (
		numaStat []meminfoMetric
		scanner  = bufio.NewScanner(r)
	)

	for scanner.Scan() {
		line := strings.TrimSpace(scanner.Text())
		if line == "" {
			continue
		}
		parts := strings.Fields(line)
		if len(parts) != 2 {
			return nil, fmt.Errorf("line scan did not return 2 fields: %s", line)
		}

		fv, err := strconv.ParseFloat(parts[1], 64)
		if err != nil {
			return nil, fmt.Errorf("invalid value in numastat: %w", err)
		}

		numaStat = append(numaStat, meminfoMetric{parts[0] + "_total", prometheus.CounterValue, nodeNumber, fv})
	}
	return numaStat, scanner.Err()
}

View on GitHub (pinned to 17ddd77c59)