k3s-io/k3s · error

failed to remove %s file: %v

Error message

failed to remove %s file: %v

What it means

writeConfigSymlink manages the well-known kubeconfig symlink (e.g. /etc/rancher/k3s/k3s.yaml -> the real kubeconfig in the data dir). It first removes any existing path; if removal fails with anything other than not-exist (permission denied, is-a-directory, read-only filesystem), this wrapped error aborts symlink setup during server start.

Source

Thrown at pkg/server/server.go:527

	splitter := func(c rune) bool {
		return c == ','
	}
	envList := []string{}
	envList = append(envList, strings.FieldsFunc(os.Getenv("NO_PROXY"), splitter)...)
	envList = append(envList, strings.FieldsFunc(os.Getenv("no_proxy"), splitter)...)
	envList = append(envList,
		".svc",
		"."+config.ClusterDomain,
		util.JoinIPNets(config.ClusterIPRanges),
		util.JoinIPNets(config.ServiceIPRanges),
	)
	os.Unsetenv("no_proxy")
	return os.Setenv("NO_PROXY", strings.Join(envList, ","))
}

func writeConfigSymlink(kubeconfig, kubeconfigSymlink string) error {
	if err := os.Remove(kubeconfigSymlink); err != nil && !os.IsNotExist(err) {
		return fmt.Errorf("failed to remove %s file: %v", kubeconfigSymlink, err)
	}
	if err := os.MkdirAll(filepath.Dir(kubeconfigSymlink), 0755); err != nil {
		return fmt.Errorf("failed to create path for symlink: %v", err)
	}
	if err := os.Symlink(kubeconfig, kubeconfigSymlink); err != nil {
		return fmt.Errorf("failed to create symlink: %v", err)
	}
	return nil
}

func isSymlink(config string) bool {
	if fi, err := os.Lstat(config); err == nil && (fi.Mode()&os.ModeSymlink == os.ModeSymlink) {
		return true
	}
	return false
}

func setNodeLabelsAndAnnotations(ctx context.Context, nodes v1.NodeClient, config *Config) error {

View on GitHub (pinned to 6ba341e396)

Solutions

1. Inspect the path: ls -la /etc/rancher/k3s/ and check whether k3s.yaml is a file, directory, or dangling symlink and who owns it.
2. Remove or rename the offending path manually (rm/rmdir), then restart k3s so the symlink is recreated.
3. Fix ownership/permissions so the k3s process can write the directory (usually run as root; adjust SELinux contexts if enforcing).
4. Prevent other tooling from placing files at the symlink path.

Example fix

# before
ls -ld /etc/rancher/k3s/k3s.yaml  # a real file owned by another user

# after
sudo rm /etc/rancher/k3s/k3s.yaml && sudo systemctl restart k3s
ls -l /etc/rancher/k3s/k3s.yaml  # -> symlink to data dir kubeconfig

// Pre-flight before start: symlink path must be a symlink or absent
if fi, err := os.Lstat(symlinkPath); err == nil {
    if fi.Mode()&os.ModeSymlink == 0 {
        log.Fatalf("%s exists and is not a symlink - remove it", symlinkPath)
    }
}

When it happens

Trigger: Server startup (or reconfiguration) where the symlink path exists but os.Remove fails: the path is a directory, the filesystem is read-only, or the process lacks permission on the parent directory because it exists under a different owner/mode.

Common situations: /etc/rancher/k3s/k3s.yaml replaced by a real file or directory owned by root while k3s runs unprivileged; hardening that made /etc/rancher read-only; SELinux/AppArmor denials on unlink; leftover artifacts from other tools writing the same path.

Related errors

• failed to create symlink: %v
• failed to create path for symlink: %v
• tar error: %v
• unhandled cgroup mode
• no bootstrap data is available to reconcile against

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/19dd97ed31db8cb4. Report an issue: GitHub.