What causes "failed to remove ns path %s: %w" in hashicorp/nomad?

Serialize teardown so only one caller removes the path (mutex or single owner of the ns lifecycle)

failed to remove ns path %s: %w

Error message

failed to remove ns path %s: %w

What it means

After successfully unmounting, UnmountNS removes the namespace file from /var/run/netns with os.Remove. If the removal fails, this error is returned. The mount is gone at this point, but the stale file remains on disk.

Source

Thrown at client/lib/nsutil/netns_linux.go:140

	wg.Wait()

	if err != nil {
		return nil, fmt.Errorf("failed to create namespace: %v", err)
	}

	return GetNS(nsPath)
}

// UnmountNS unmounts the NS held by the netns object
func UnmountNS(nsPath string) error {
	// Only unmount if it's been bind-mounted (don't touch namespaces in /proc...)
	if strings.HasPrefix(nsPath, NetNSRunDir) {
		if err := unix.Unmount(nsPath, unix.MNT_DETACH); err != nil {
			return fmt.Errorf("failed to unmount NS: at %s: %w", nsPath, err)
		}

		if err := os.Remove(nsPath); err != nil {
			return fmt.Errorf("failed to remove ns path %s: %w", nsPath, err)
		}
	}

	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

Serialize teardown so only one caller removes the path (mutex or single owner of the ns lifecycle)
Ignore ENOENT as success if concurrent removal is possible (check errors.Is(err, fs.ErrNotExist))
Ensure the process user has write permission on /run/netns
Verify /run is writable (not mounted read-only)

Example fix

// before: races report ENOENT
if err := os.Remove(nsPath); err != nil {
    return fmt.Errorf("failed to remove ns path %s: %w", nsPath, err)
}
// after (caller-side tolerance)
err := nsutil.UnmountNS(nsPath)
if err != nil && !errors.Is(err, fs.ErrNotExist) {
    return err
}

Defensive patterns

Strategy: try-catch

Validate before calling

if fi, err := os.Stat(nsPath); err != nil || fi.IsDir() {
    return fmt.Errorf("invalid ns path %s", nsPath)
}

Try / catch

err := nsutil.UnmountNS(nsPath)
if err != nil && errors.Is(err, fs.ErrNotExist) {
    return nil // already removed by a concurrent teardown
}
if err != nil {
    return fmt.Errorf("stale ns file at %s: %w", nsPath, err)
}

Prevention

Serialize DestroyNetwork calls per namespace (single owner/mutex)
Treat ENOENT during cleanup as success
Ensure the process user owns /run/netns or has write access
Keep /run writable; alert on read-only remounts

When it happens

Trigger: unix.Unmount succeeded but os.Remove(nsPath) fails during UnmountNS (via DestroyNetwork) — EACCES due to directory permissions, ENOENT if the file vanished concurrently, or read-only filesystem.

Common situations: Two concurrent DestroyNetwork calls racing on the same nsPath; /run mounted read-only after a container state change; wrong ownership/permissions on /run/netns.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/17118c97cbce327b. Report an issue: GitHub.