What causes "Failed to create image import watcher:" in k3s-io/k3s?

Ensure the watched directory exists: mkdir -p /var/lib/rancher/k3s/agent/images (and check cfg.Images custom path if --node-... image dir flags were set)

k3s-io/k3s · critical

Failed to create image import watcher:

Error message

Failed to create image import watcher:

What it means

k3s airgap image-import watcher (pkg/agent/containerd/watcher.go): mustCreateWatcher wraps createWatcher, which does fsnotify.NewWatcher() and watcher.Add(path) on the directory containing the airgap image list (filepath.Dir(cfg.Images), default /var/lib/rancher/k3s/agent/images). Any failure panics with this message. Typical causes: inotify instance/watch limits exhausted (ENOSPC), the watched directory missing, or permission denial on the directory.

Source

Thrown at pkg/agent/containerd/watcher.go:54

}

func createWatcher(path string) (*fsnotify.Watcher, error) {
	watcher, err := fsnotify.NewWatcher()
	if err != nil {
		return nil, err
	}

	if err := watcher.Add(path); err != nil {
		return nil, err
	}

	return watcher, nil
}

func mustCreateWatcher(path string) *fsnotify.Watcher {
	watcher, err := createWatcher(path)
	if err != nil {
		panic("Failed to create image import watcher:" + err.Error())
	}
	return watcher
}

func isFileSupported(path string) bool {
	for _, ext := range append(tarfile.SupportedExtensions, ".txt") {
		if strings.HasSuffix(path, ext) {
			return true
		}
	}

	return false
}

// runWorkerForImages connects to containerd and calls processNextEventForImages to process items from the workqueue.
// This blocks until the workqueue is shut down.
func (w *watchqueue) runWorkerForImages(ctx context.Context) {
	// create the connections to not create every time when processing a event

View on GitHub (pinned to 6ba341e396)

Solutions

Ensure the watched directory exists: mkdir -p /var/lib/rancher/k3s/agent/images (and check cfg.Images custom path if --node-... image dir flags were set)
Raise inotify limits: sysctl fs.inotify.max_user_instances=512 (or higher) and fs.inotify.max_user_watches=1048576, then persist in sysctl.d and restart k3s
Audit other processes holding inotify watches (e.g. busy sidecars, filebeat, IDE servers) and reduce them, or raise the system-wide limits instead of per-process retries
Verify permissions and writability of the k3s dataDir; repair read-only filesystems (disk full, NFS/overlay quirks) and restart the agent

Example fix

// before
func mustCreateWatcher(path string) *fsnotify.Watcher {
    watcher, err := createWatcher(path)
    if err != nil {
        panic("Failed to create image import watcher:" + err.Error())
    }
    return watcher
}

// after
func mustCreateWatcher(path string) *fsnotify.Watcher {
    if err := os.MkdirAll(path, 0o755); err != nil {
        panic("Failed to create image import watcher dir:" + err.Error())
    }
    watcher, err := createWatcher(path)
    if err != nil {
        panic("Failed to create image import watcher:" + err.Error())
    }
    return watcher
}

Defensive patterns

Strategy: validation

Validate before calling

watchDir := filepath.Dir(cfg.Images)
if err := os.MkdirAll(watchDir, 0o755); err != nil {
    return fmt.Errorf("cannot ensure image import dir %s: %w", watchDir, err)
}
// fail fast with a clear message when inotify limits are the real cause
if instances, err := strconv.Atoi(firstLine("/proc/sys/fs/inotify/max_user_instances")); err == nil && instances < 128 {
    logrus.Warnf("fs.inotify.max_user_instances=%d is low; image import watcher may fail", instances)
}

Try / catch

watcher, err := createWatcher(path)
if err != nil {
    // do not panic in caller context: classify ENOSPC (inotify limits) vs ENOENT (missing dir)
    if errors.Is(err, syscall.ENOSPC) {
        return fmt.Errorf("inotify limits exhausted; raise fs.inotify.max_user_instances/max_user_watches: %w", err)
    }
    if errors.Is(err, fs.ErrNotExist) {
        return fmt.Errorf("image import dir %s missing: %w", path, err)
    }
    return err
}

Prevention

Persist sysctl fs.inotify.max_user_instances=512 and fs.inotify.max_user_watches=1048576 on nodes that run k3s plus other watch-heavy software
Keep the airgap images directory present and writable even when empty
Monitor for ENOSPC in agent logs as an early signal before the panic path is hit
When embedding k3s, prefer calling createWatcher and propagating the error instead of mustCreateWatcher so startup can degrade gracefully

When it happens

Trigger: Starting k3s (or re-arming the watcher at watcher.go:320/336 after a watch error) on a node where fs.inotify.max_user_instances/max_user_watches are exhausted by other containers; the agent images directory was deleted or never created; read-only or badly permissioned dataDir.

Common situations: Heavily loaded hosts running many containers/agents that consume inotify instances; airgap installs where /var/lib/rancher/k3s/agent/images was removed after setup; security-hardened nodes restricting inotify; disk-full or read-only filesystem conditions.

Related errors

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/d7515ca7466ec0ea. Report an issue: GitHub.