What causes "failed to unmount NS: at %s: %w" in hashicorp/nomad?

Ensure no processes remain in the namespace before destroying it (kill containers/test processes first)

failed to unmount NS: at %s: %w

Error message

failed to unmount NS: at %s: %w

What it means

UnmountNS tears down a namespace mount created by NewNS by calling umount2 with MNT_DETACH on the path. If the unmount syscall fails, this error is returned. It means the bind mount under /var/run/netns could not be detached, typically because it is busy or the caller lacks privileges.

Source

Thrown at client/lib/nsutil/netns_linux.go:136

		if err != nil {
			err = fmt.Errorf("failed to bind mount ns at %s: %v", nsPath, err)
		}
	})()
	wg.Wait()

	if err != nil {
		return nil, fmt.Errorf("failed to create namespace: %v", err)
	}

	return GetNS(nsPath)
}

// UnmountNS unmounts the NS held by the netns object
func UnmountNS(nsPath string) error {
	// Only unmount if it's been bind-mounted (don't touch namespaces in /proc...)
	if strings.HasPrefix(nsPath, NetNSRunDir) {
		if err := unix.Unmount(nsPath, unix.MNT_DETACH); err != nil {
			return fmt.Errorf("failed to unmount NS: at %s: %w", nsPath, err)
		}

		if err := os.Remove(nsPath); err != nil {
			return fmt.Errorf("failed to remove ns path %s: %w", nsPath, err)
		}
	}

	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

Ensure no processes remain in the namespace before destroying it (kill containers/test processes first)
Check for leftover mounts: grep netns /proc/mounts and umount manually if needed
If EINVAL because it is not a mountpoint, treat the path as already unmounted and just remove the file
Run with sufficient privileges (CAP_SYS_ADMIN) for umount

Example fix

// before: double teardown triggers EINVAL/EBUSY
err := nsutil.UnmountNS(ns.Path())
// after: guard against double-unmount
if _, statErr := os.Stat(ns.Path()); statErr == nil {
    if out, err := exec.Command("sh", "-c", "grep -qs "+ns.Path()+" /proc/mounts").Output(); err == nil && len(out) > 0 {
        err = nsutil.UnmountNS(ns.Path())
    }
}

Defensive patterns

Strategy: try-catch

Validate before calling

out, _ := os.ReadFile("/proc/mounts")
if !strings.Contains(string(out), "/run/netns/") {
    // nothing mounted at this path; skip unmount
}

Type guard

func isMountedUnderRunNetns(nsPath string) bool {
    return strings.HasPrefix(nsPath, "/run/netns/")
}

Try / catch

err := nsutil.UnmountNS(nsPath)
if err != nil {
    var errno unix.Errno
    if errors.As(err, &errno) && (errno == unix.EINVAL || errno == unix.EBUSY) {
        log.Printf("ns mount busy/absent at %s, cleaning up processes and retrying", nsPath)
    }
    return err
}

Prevention

Terminate all processes inside the namespace before calling UnmountNS
Make network teardown idempotent (tolerate not-a-mountpoint)
Run teardown with CAP_SYS_ADMIN
Clean stale entries in /run/netns during startup

When it happens

Trigger: UnmountNS (via DestroyNetwork) is called with a path under /var/run/netns and unix.Unmount(nsPath, MNT_DETACH) returns an error — EBUSY when the mount is in use, EPERM without privileges, EINVAL when the path is not a mountpoint.

Common situations: Processes (veth handles, test binaries) still running inside the namespace keeping the mount busy; calling DestroyNetwork twice with a stale path; path passed is not actually a mountpoint (already unmounted).

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/de9d60da224a1651. Report an issue: GitHub.