What causes "couldn't connect netlink: %w" in prometheus/node_exporter?

Check seccomp/apparmor logs for blocked socket(AF_NETLINK) calls and allow them or use a permissive profile

couldn't connect netlink

Error message

couldn't connect netlink: %w

What it means

getTCPStats opens a netlink socket with netlink.Dial(syscall.NETLINK_INET_DIAG, nil) to issue sock_diag queries. If the socket cannot be created or connected, this wrapped error is returned. Without this socket, TCP state counting (ESTABLISHED, LISTEN, etc.) cannot be performed at all.

Solutions

Check seccomp/apparmor logs for blocked socket(AF_NETLINK) calls and allow them or use a permissive profile
Raise fd limits if EMFILE/ENFILE is the wrapped cause (ulimit -n, systemd LimitNOFILE)
Run the exporter with sufficient privileges/capabilities in the container runtime
Fall back to dropping --collector.tcpstat if the environment cannot permit netlink

Example fix

// gVisor runsc blocks netlink by default
// before: runsc seccomp default denies AF_NETLINK
// after: use --profile=custom allowing socket(AF_NETLINK, SOCK_RAW, NETLINK_INET_DIAG) or run without gVisor

Defensive patterns

Strategy: try-catch

Validate before calling

conn, err := netlink.Dial(syscall.NETLINK_INET_DIAG, nil)
if err != nil { /* netlink unusable here — skip tcpstat */ } else { conn.Close() }

Try / catch

if err := coll.Update(ch); err != nil {
    var nerr net.Error
    if errors.As(err, &nerr) || errors.Is(err, os.ErrPermission) {
        log.Warn("netlink dial blocked; falling back")
    }
}

Prevention

Run seccomp/apparmor profiles that allow AF_NETLINK sockets
Raise RLIMIT_NOFILE for the exporter under heavy load
Avoid runtimes (gVisor, hardened sandboxes) that block netlink on monitoring hosts
Keep the tcpstat collector disabled where netlink is not permitted

When it happens

Trigger: netlink.Dial(NETLINK_INET_DIAG) fails: no permission to open netlink sockets (SELinux/seccomp/AppArmor), socket exhaustion (EMFILE/ENFILE), or the kernel lacks NETLINK_INET_DIAG support.

Common situations: Strict container runtimes (gVisor, Kata, minimal seccomp profiles) that block socket(AF_NETLINK); fd limits exhausted under load; security modules denying netlink usage for the node_exporter user.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

AI-assisted analysis of prometheus/node_exporter@17ddd77c59 (2026-09-07). Data as JSON: /api/errors/55ce8fc7dc5a8e20. Report an issue: GitHub.

Appendix: source

Thrown at collector/tcpstat_linux.go:162

			tcpStats[st] += value
		}
	}

	for st, value := range tcpStats {
		ch <- c.desc.mustNewConstMetric(value, st.String())
	}

	return nil
}

func getTCPStats(family uint8) (map[tcpConnectionState]float64, error) {
	const TCPFAll = 0xFFF
	const InetDiagInfo = 2
	const SockDiagByFamily = 20

	conn, err := netlink.Dial(syscall.NETLINK_INET_DIAG, nil)
	if err != nil {
		return nil, fmt.Errorf("couldn't connect netlink: %w", err)
	}
	defer conn.Close()

	msg := netlink.Message{
		Header: netlink.Header{
			Type:  SockDiagByFamily,
			Flags: syscall.NLM_F_REQUEST | syscall.NLM_F_DUMP,
		},
		Data: (&InetDiagReqV2{
			Family:   family,
			Protocol: syscall.IPPROTO_TCP,
			States:   TCPFAll,
			Ext:      0 | 1<<(InetDiagInfo-1),
		}).Serialize(),
	}

	messages, err := conn.Execute(msg)
	if err != nil {

View on GitHub (pinned to 17ddd77c59)