{"record":{"id":"0e7b60505e60bcef","repo":"kubernetes/kops","slug":"error-reading-var-log-v","errorCode":null,"errorMessage":"error reading /var/log: %v","messagePattern":"error reading /var/log: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"pkg/dump/dumper.go","lineNumber":465,"sourceCode":"\t\t}\n\t\tif err := n.shellToFile(ctx, \"sudo ss -tanp state all '( sport = :179 or dport = :179 )'\", filepath.Join(n.dir, \"bgp-sockets.log\")); err != nil {\n\t\t\terrors = append(errors, err)\n\t\t}\n\t\t// The conntrack CLI is absent from most node images, so fall back to the kernel\n\t\t// table, which is populated whenever kube-proxy is running. It cannot be guarded\n\t\t// with a test: /proc/net/nf_conntrack is 0440 root:root and this shell is not\n\t\t// root, so only the privileged read itself can tell us whether it is there.\n\t\tconst conntrack179 = `if command -v conntrack &> /dev/null; then sudo conntrack -L -p tcp --dport 179; ` +\n\t\t\t`else sudo grep -E 'sport=179|dport=179' /proc/net/nf_conntrack 2>/dev/null || true; fi`\n\t\tif err := n.shellToFile(ctx, conntrack179, filepath.Join(n.dir, \"bgp-conntrack.log\")); err != nil {\n\t\t\terrors = append(errors, err)\n\t\t}\n\t}\n\n\t// Capture any file logs where the files exist\n\tfileList, err := n.findFiles(ctx, \"/var/log\")\n\tif err != nil {\n\t\terrors = append(errors, fmt.Errorf(\"error reading /var/log: %v\", err))\n\t}\n\tfor _, name := range n.dumper.files {\n\t\tprefix := \"/var/log/\" + name + \".log\"\n\t\tfor _, f := range fileList {\n\t\t\tif !strings.HasPrefix(f, prefix) {\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tif err := n.shellToFile(ctx, \"sudo cat '\"+strings.ReplaceAll(f, \"'\", \"'\\\\''\")+\"'\", filepath.Join(n.dir, strings.ReplaceAll(strings.TrimPrefix(f, \"/var/log/\"), \"/\", \"_\"))); err != nil {\n\t\t\t\terrors = append(errors, err)\n\t\t\t}\n\t\t}\n\t}\n\n\tfor _, selector := range n.dumper.podSelectors {\n\t\tkv := strings.Split(selector, \"=\")\n\t\tlogFile := fmt.Sprintf(\"%v.log\", kv[len(kv)-1])\n\t\tif err := n.shellToFile(ctx, \"if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \\\"\"+selector+\"\\\"; fi\", filepath.Join(n.dir, logFile)); err != nil {\n\t\t\terrors = append(errors, err)","sourceCodeStart":447,"sourceCodeEnd":483,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/pkg/dump/dumper.go#L447-L483","documentation":"Accumulated by logDumperNode.dump (pkg/dump/dumper.go:465) when n.findFiles(ctx, \"/var/log\") fails. This enumerates files under /var/log on the remote node via `sudo find /var/log -type f -print0`; without it, none of the dumper.files entries (kubelet.log, etc.) can be matched and copied. Like other per-node dump errors it is logged via klog.Warningf and does not fail the overall node dump.","triggerScenarios":"findFiles' ExecPiped of `sudo find /var/log -type f -print0` returns an error: the 1-minute nodeDumpTimeout expires on a node with a huge /var/log tree, the SSH session dies mid-command, or the remote shell exits non-zero (e.g. sudo denied, /var/log missing/corrupt filesystem).","commonSituations":"Node with multi-GB /var/log directories blowing the default 1-minute timeout; disk-full or read-only root filesystem on the node; SSH disconnect during large dumps; container-optimized images where /var/log is a symlink loop or mounted oddly.","solutions":["Raise --node-dump-timeout (large clusters need more than the 60s default since dumps are sequential).","SSH to the node and run `sudo find /var/log -type f` manually to reproduce and see the underlying failure.","Check node health: `df -h`, `dmesg` for read-only/failed filesystems preventing the find from completing.","Retry the dump once the node has settled (e.g. after an upgrade or heavy log rotation finishes).","If only /var/log is affected, other artifacts (journal, iptables) may still have been captured — check the artifacts dir before re-dumping."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"// ensure the remote dir exists and find completes quickly\nerr := client.ExecPiped(ctx, \"test -d /var/log && sudo find /var/log -type f | head -1\", io.Discard, io.Discard)\nif err != nil {\n    klog.Warningf(\"/var/log pre-check failed; file-log capture will be skipped: %v\", err)\n}","typeGuard":null,"tryCatchPattern":"fileList, err := n.findFiles(ctx, \"/var/log\")\nif err != nil {\n    klog.Warningf(\"file-log capture skipped for %s: %v\", n.dir, err)\n    fileList = nil\n}","preventionTips":["Raise --node-dump-timeout when nodes have very large /var/log trees.","Monitor node disk health (read-only mounts fail find).","Re-run dumps after node upgrades rather than during them."],"tags":["ssh","node-dump","filesystem","go"],"backgroundTag":"remote-command-failed","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T07:17:12.445Z"}