hashicorp/nomad · error

failed to remove alloc dir %q: %w

Error message

failed to remove alloc dir %q: %w

What it means

AllocDir.Destroy() removes the whole allocation directory tree with os.RemoveAll. If that syscall fails for any reason, the underlying OS error is wrapped with the alloc dir path into this message and appended to a multierror (along with any UnmountAll failure) returned by Destroy.

Source

Thrown at client/allocdir/alloc_dir.go:330

			if err := os.Rename(otherTaskLocal, localDir); err != nil {
				return fmt.Errorf("error moving task %q local dir: %w", task.Name, err)
			}
		}
	}

	return nil
}

// Destroy tears down previously build directory structure.
func (a *AllocDir) Destroy() error {
	// Unmount all mounted shared alloc dirs.
	mErr := new(multierror.Error)
	if err := a.UnmountAll(); err != nil {
		mErr = multierror.Append(mErr, err)
	}

	if err := os.RemoveAll(a.AllocDir); err != nil {
		mErr = multierror.Append(mErr, fmt.Errorf("failed to remove alloc dir %q: %w", a.AllocDir, err))
	}

	// Unset built since the alloc dir has been destroyed.
	a.mu.Lock()
	a.built = false
	a.mu.Unlock()
	return mErr.ErrorOrNil()
}

// UnmountAll linked/mounted directories in task dirs.
func (a *AllocDir) UnmountAll() error {
	a.mu.RLock()
	defer a.mu.RUnlock()

	mErr := new(multierror.Error)
	for _, dir := range a.TaskDirs {
		if err := dir.Unmount(); err != nil {
			mErr = multierror.Append(mErr, err)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Ensure all tasks are stopped and UnmountAll has unmounted every mount before Destroy
2. Check for leaked processes holding mounts in the alloc dir (lsof / findmnt) and kill or unmount them
3. Verify the nomad user has write permission on the alloc dir and its parent and the filesystem is not read-only
4. Manually remove the leftover directory (rm -rf) and investigate the wrapped OS error for the root cause

Example fix

// before
if err := allocDir.Destroy(); err != nil {
    logger.Error("destroy failed", "err", err)
}
// after
if err := allocDir.UnmountAll(); err != nil {
    logger.Warn("unmount leaked mounts", "err", err)
}
if err := allocDir.Destroy(); err != nil {
    logger.Error("destroy failed", "err", err)
}

if _, err := os.Stat(allocDir.AllocDir); err == nil {
    if err := allocDir.UnmountAll(); err != nil {
        logger.Warn("unmount incomplete", "err", err)
    }
}

if err := allocDir.Destroy(); err != nil {
    var pathErr *os.PathError
    if errors.As(err, &pathErr) && errors.Is(pathErr.Err, unix.EBUSY) {
        // leaked mount: find and unmount, then retry Destroy
    }
    logger.Error("alloc dir destroy failed", "err", err)
}

When it happens

Trigger: Calling AllocDir.Destroy() when os.RemoveAll(a.AllocDir) fails: files inside the tree still held open or mounted by a running task, read-only filesystem, or permission denied on a subdirectory owned by another user.

Common situations: Tearing down an allocation whose tasks or other processes still have mounts/files open in the alloc dir; running the Nomad client with insufficient privileges; leftover bind mounts from leaked task processes preventing removal.

Related errors

• Failed to make the alloc directory %v: %w
• Failed to lookup nobody user: %v
• Failed to create task mount directory: %v
• Failed to mount task dir: %v
• Failed to mount alloc dir: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/ad1a650b0236b088. Report an issue: GitHub.