{"record":{"id":"d7515ca7466ec0ea","repo":"k3s-io/k3s","slug":"failed-to-create-image-import-watcher","errorCode":null,"errorMessage":"Failed to create image import watcher:","messagePattern":"Failed to create image import watcher:","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"pkg/agent/containerd/watcher.go","lineNumber":54,"sourceCode":"}\n\nfunc createWatcher(path string) (*fsnotify.Watcher, error) {\n\twatcher, err := fsnotify.NewWatcher()\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tif err := watcher.Add(path); err != nil {\n\t\treturn nil, err\n\t}\n\n\treturn watcher, nil\n}\n\nfunc mustCreateWatcher(path string) *fsnotify.Watcher {\n\twatcher, err := createWatcher(path)\n\tif err != nil {\n\t\tpanic(\"Failed to create image import watcher:\" + err.Error())\n\t}\n\treturn watcher\n}\n\nfunc isFileSupported(path string) bool {\n\tfor _, ext := range append(tarfile.SupportedExtensions, \".txt\") {\n\t\tif strings.HasSuffix(path, ext) {\n\t\t\treturn true\n\t\t}\n\t}\n\n\treturn false\n}\n\n// runWorkerForImages connects to containerd and calls processNextEventForImages to process items from the workqueue.\n// This blocks until the workqueue is shut down.\nfunc (w *watchqueue) runWorkerForImages(ctx context.Context) {\n\t// create the connections to not create every time when processing a event","sourceCodeStart":36,"sourceCodeEnd":72,"githubUrl":"https://github.com/k3s-io/k3s/blob/6ba341e396edc16b8dcae978a7c5e3ac7ee5606e/pkg/agent/containerd/watcher.go#L36-L72","documentation":"k3s airgap image-import watcher (pkg/agent/containerd/watcher.go): mustCreateWatcher wraps createWatcher, which does fsnotify.NewWatcher() and watcher.Add(path) on the directory containing the airgap image list (filepath.Dir(cfg.Images), default /var/lib/rancher/k3s/agent/images). Any failure panics with this message. Typical causes: inotify instance/watch limits exhausted (ENOSPC), the watched directory missing, or permission denial on the directory.","triggerScenarios":"Starting k3s (or re-arming the watcher at watcher.go:320/336 after a watch error) on a node where fs.inotify.max_user_instances/max_user_watches are exhausted by other containers; the agent images directory was deleted or never created; read-only or badly permissioned dataDir.","commonSituations":"Heavily loaded hosts running many containers/agents that consume inotify instances; airgap installs where /var/lib/rancher/k3s/agent/images was removed after setup; security-hardened nodes restricting inotify; disk-full or read-only filesystem conditions.","solutions":["Ensure the watched directory exists: mkdir -p /var/lib/rancher/k3s/agent/images (and check cfg.Images custom path if --node-... image dir flags were set)","Raise inotify limits: sysctl fs.inotify.max_user_instances=512 (or higher) and fs.inotify.max_user_watches=1048576, then persist in sysctl.d and restart k3s","Audit other processes holding inotify watches (e.g. busy sidecars, filebeat, IDE servers) and reduce them, or raise the system-wide limits instead of per-process retries","Verify permissions and writability of the k3s dataDir; repair read-only filesystems (disk full, NFS/overlay quirks) and restart the agent"],"exampleFix":"// before\nfunc mustCreateWatcher(path string) *fsnotify.Watcher {\n    watcher, err := createWatcher(path)\n    if err != nil {\n        panic(\"Failed to create image import watcher:\" + err.Error())\n    }\n    return watcher\n}\n\n// after\nfunc mustCreateWatcher(path string) *fsnotify.Watcher {\n    if err := os.MkdirAll(path, 0o755); err != nil {\n        panic(\"Failed to create image import watcher dir:\" + err.Error())\n    }\n    watcher, err := createWatcher(path)\n    if err != nil {\n        panic(\"Failed to create image import watcher:\" + err.Error())\n    }\n    return watcher\n}","handlingStrategy":"validation","validationCode":"watchDir := filepath.Dir(cfg.Images)\nif err := os.MkdirAll(watchDir, 0o755); err != nil {\n    return fmt.Errorf(\"cannot ensure image import dir %s: %w\", watchDir, err)\n}\n// fail fast with a clear message when inotify limits are the real cause\nif instances, err := strconv.Atoi(firstLine(\"/proc/sys/fs/inotify/max_user_instances\")); err == nil && instances < 128 {\n    logrus.Warnf(\"fs.inotify.max_user_instances=%d is low; image import watcher may fail\", instances)\n}","typeGuard":null,"tryCatchPattern":"watcher, err := createWatcher(path)\nif err != nil {\n    // do not panic in caller context: classify ENOSPC (inotify limits) vs ENOENT (missing dir)\n    if errors.Is(err, syscall.ENOSPC) {\n        return fmt.Errorf(\"inotify limits exhausted; raise fs.inotify.max_user_instances/max_user_watches: %w\", err)\n    }\n    if errors.Is(err, fs.ErrNotExist) {\n        return fmt.Errorf(\"image import dir %s missing: %w\", path, err)\n    }\n    return err\n}","preventionTips":["Persist sysctl fs.inotify.max_user_instances=512 and fs.inotify.max_user_watches=1048576 on nodes that run k3s plus other watch-heavy software","Keep the airgap images directory present and writable even when empty","Monitor for ENOSPC in agent logs as an early signal before the panic path is hit","When embedding k3s, prefer calling createWatcher and propagating the error instead of mustCreateWatcher so startup can degrade gracefully"],"tags":["fsnotify","inotify","containerd","airgap","panic","k3s"],"backgroundTag":null,"analyzedSha":"6ba341e396edc16b8dcae978a7c5e3ac7ee5606e","analyzedAt":"2026-08-15T16:27:54.286Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}