What causes "couldn't get links: %w" in prometheus/node_exporter?

Retry the scrape; transient netlink dump failures often resolve on the next scrape.

couldn't get links

Error message

couldn't get links: %w

What it means

After successfully connecting to rtnetlink, the network_route collector lists all network links via conn.Link.List(). If the netlink dump or its parsing fails, Update returns this wrapped error and no route metrics are emitted for the scrape.

Solutions

Retry the scrape; transient netlink dump failures often resolve on the next scrape.
Reduce interface churn or count on the host (clean up orphaned veth devices).
Update mdlayher/netlink, mdlayher/rtnetlink, and node_exporter to get buffer/parse fixes.
Check kernel logs for netlink errors; if persistent with many links, report upstream.

Example fix

// before
links, err := conn.Link.List()
if err != nil {
	return fmt.Errorf("couldn't get links: %w", err)
}
// after (one retry on transient failure)
links, err := conn.Link.List()
if err != nil {
	links, err = conn.Link.List()
	if err != nil {
		return fmt.Errorf("couldn't get links: %w", err)
	}
}

Defensive patterns

Strategy: retry

Validate before calling

null

Type guard

null

Try / catch

// Retry transient netlink dump failures once before failing the scrape
links, err := conn.Link.List()
if err != nil {
	time.Sleep(50 * time.Millisecond)
	if links, err = conn.Link.List(); err != nil {
		return fmt.Errorf("couldn't get links: %w", err)
	}
}

Prevention

Keep mdlayher/netlink and mdlayher/rtnetlink updated for dump robustness fixes.
Watch for ENOBUFS-style failures on hosts with thousands of interfaces.
Reduce veth/interface churn on busy container hosts.
Alert on repeated node_scrape_collector_success{collector="network_route"}==0 rather than single failures.

When it happens

Trigger: conn.Link.List() fails on netlink dump errors (socket closed mid-dump, buffer too small for a very large number of links, kernel/netlink message parse failure from mdlayher/netlink).

Common situations: Transient netlink buffer overflows on hosts with thousands of interfaces (many veth/containers); kernel bugs; intermittent netlink EBUSY/ENOBUFS under heavy interface churn.

Related errors

AI-assisted analysis of prometheus/node_exporter@17ddd77c59 (2026-09-07). Data as JSON: /api/errors/164068af7114f5cd. Report an issue: GitHub.

Appendix: source

Thrown at collector/network_route_linux.go:70

	return &networkRouteCollector{
		routeInfoDesc: routeInfoDesc,
		routesDesc:    routesDesc,
		logger:        logger,
	}, nil
}

func (n networkRouteCollector) Update(ch chan<- prometheus.Metric) error {
	deviceRoutes := make(map[string]int)

	conn, err := rtnetlink.Dial(nil)
	if err != nil {
		return fmt.Errorf("couldn't connect rtnetlink: %w", err)
	}
	defer conn.Close()

	links, err := conn.Link.List()
	if err != nil {
		return fmt.Errorf("couldn't get links: %w", err)
	}

	routes, err := conn.Route.List()
	if err != nil {
		return fmt.Errorf("couldn't get routes: %w", err)
	}

	for _, route := range routes {
		if route.Type != unix.RTA_DST {
			continue
		}
		if len(route.Attributes.Multipath) != 0 {
			for _, nextHop := range route.Attributes.Multipath {
				ifName := ""
				for _, link := range links {
					if link.Index == nextHop.Hop.IfIndex {
						ifName = link.Attributes.Name
						break

View on GitHub (pinned to 17ddd77c59)