kubernetes/kops · warning

error reading /var/log: %v

Error message

error reading /var/log: %v

What it means

Accumulated by logDumperNode.dump (pkg/dump/dumper.go:465) when n.findFiles(ctx, "/var/log") fails. This enumerates files under /var/log on the remote node via `sudo find /var/log -type f -print0`; without it, none of the dumper.files entries (kubelet.log, etc.) can be matched and copied. Like other per-node dump errors it is logged via klog.Warningf and does not fail the overall node dump.

Source

Thrown at pkg/dump/dumper.go:465

		}
		if err := n.shellToFile(ctx, "sudo ss -tanp state all '( sport = :179 or dport = :179 )'", filepath.Join(n.dir, "bgp-sockets.log")); err != nil {
			errors = append(errors, err)
		}
		// The conntrack CLI is absent from most node images, so fall back to the kernel
		// table, which is populated whenever kube-proxy is running. It cannot be guarded
		// with a test: /proc/net/nf_conntrack is 0440 root:root and this shell is not
		// root, so only the privileged read itself can tell us whether it is there.
		const conntrack179 = `if command -v conntrack &> /dev/null; then sudo conntrack -L -p tcp --dport 179; ` +
			`else sudo grep -E 'sport=179|dport=179' /proc/net/nf_conntrack 2>/dev/null || true; fi`
		if err := n.shellToFile(ctx, conntrack179, filepath.Join(n.dir, "bgp-conntrack.log")); err != nil {
			errors = append(errors, err)
		}
	}

	// Capture any file logs where the files exist
	fileList, err := n.findFiles(ctx, "/var/log")
	if err != nil {
		errors = append(errors, fmt.Errorf("error reading /var/log: %v", err))
	}
	for _, name := range n.dumper.files {
		prefix := "/var/log/" + name + ".log"
		for _, f := range fileList {
			if !strings.HasPrefix(f, prefix) {
				continue
			}
			if err := n.shellToFile(ctx, "sudo cat '"+strings.ReplaceAll(f, "'", "'\\''")+"'", filepath.Join(n.dir, strings.ReplaceAll(strings.TrimPrefix(f, "/var/log/"), "/", "_"))); err != nil {
				errors = append(errors, err)
			}
		}
	}

	for _, selector := range n.dumper.podSelectors {
		kv := strings.Split(selector, "=")
		logFile := fmt.Sprintf("%v.log", kv[len(kv)-1])
		if err := n.shellToFile(ctx, "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \""+selector+"\"; fi", filepath.Join(n.dir, logFile)); err != nil {
			errors = append(errors, err)

View on GitHub (pinned to 4c8573c808)

Solutions

1. Raise --node-dump-timeout (large clusters need more than the 60s default since dumps are sequential).
2. SSH to the node and run `sudo find /var/log -type f` manually to reproduce and see the underlying failure.
3. Check node health: `df -h`, `dmesg` for read-only/failed filesystems preventing the find from completing.
4. Retry the dump once the node has settled (e.g. after an upgrade or heavy log rotation finishes).
5. If only /var/log is affected, other artifacts (journal, iptables) may still have been captured — check the artifacts dir before re-dumping.

// ensure the remote dir exists and find completes quickly
err := client.ExecPiped(ctx, "test -d /var/log && sudo find /var/log -type f | head -1", io.Discard, io.Discard)
if err != nil {
    klog.Warningf("/var/log pre-check failed; file-log capture will be skipped: %v", err)
}

fileList, err := n.findFiles(ctx, "/var/log")
if err != nil {
    klog.Warningf("file-log capture skipped for %s: %v", n.dir, err)
    fileList = nil
}

When it happens

Trigger: findFiles' ExecPiped of `sudo find /var/log -type f -print0` returns an error: the 1-minute nodeDumpTimeout expires on a node with a huge /var/log tree, the SSH session dies mid-command, or the remote shell exits non-zero (e.g. sudo denied, /var/log missing/corrupt filesystem).

Common situations: Node with multi-GB /var/log directories blowing the default 1-minute timeout; disk-full or read-only root filesystem on the node; SSH disconnect during large dumps; container-optimized images where /var/log is a symlink loop or mounted oddly.

Related errors

• error listing %q: %v
• error listing systemd services: %v
• error listing /etc/containerd: %v
• error listing systemd units: %v
• error creating file %q: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/0e7b60505e60bcef. Report an issue: GitHub.