hashicorp/nomad · error

unable to remove existing unix socket: %w

Error message

unable to remove existing unix socket: %w

What it means

Nomad's Consul HTTP socket hook creates a unix socket in the alloc dir so tasks can reach Consul via a proxied socket. maybeRemoveOldSocket stats the socket path and removes a stale socket left from a previous allocation run. This error wraps the os.Remove failure when the file exists but cannot be deleted, typically due to filesystem permissions or a busy mount.

Source

Thrown at client/allocrunner/consul_http_sock_hook.go:270

	// if proxy was never run, no need to wait before shutdown
	if !p.runOnce {
		return nil
	}

	select {
	case <-p.doneCh:
	case <-time.After(socketProxyStopWaitTime):
		return errSocketProxyTimeout
	}

	return nil
}

func maybeRemoveOldSocket(socketPath string) error {
	_, err := os.Stat(socketPath)
	if err == nil {
		if err = os.Remove(socketPath); err != nil {
			return fmt.Errorf("unable to remove existing unix socket: %w", err)
		}
	}
	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check ownership/permissions of the alloc dir socket path and fix them (chown to the nomad user) or remove the stale file manually
2. Ensure the parent directory is writable by the Nomad agent user and not mounted read-only
3. Restart the Nomad client or stop the task so the socket is not in use, then retry the allocation
4. As a last resort, clean the affected alloc dir (nomad system reap / remove client data dir for that alloc)

Example fix

// before
# ls -l <alloc_dir>/consul_http.sock  -> owned by root, nomad can't unlink
// after
# chown nomad:nomad <alloc_dir> && rm -f <alloc_dir>/consul_http.sock
# nomad alloc stop <alloc_id>  # reschedules and recreates socket

const st = await fs.promises.stat(socketPath).catch(e => e.code === 'ENOENT' ? null : Promise.reject(e));
if (st) await fs.promises.access(path.dirname(socketPath), fs.constants.W_OK | fs.constants.X_OK);

try {
  await fs.promises.rm(socketPath, { force: true });
} catch (err) {
  if (err.code === 'EACCES' || err.code === 'EPERM' || err.code === 'EBUSY') {
    // surface actionable hint: check alloc dir ownership / mount state
    throw new Error(`cannot remove stale socket ${socketPath}: ${err.message} (check perms/mounts)`);
  }
  throw err;
}

When it happens

Trigger: os.Stat(socketPath) succeeds (file/socket exists) and os.Remove(socketPath) fails with e.g. EACCES, EPERM, or EBUSY during alloc runner hook execution.

Common situations: Alloc data dir owned by a different user after a Nomad upgrade or user change; read-only or full host volume backing the alloc dir; the socket path is a mount point that cannot be unlinked; leftover sockets after an unclean node shutdown on bind-mounted volumes.

Related errors

• plugin not executable
• Chmod(%v) failed: %w
• Couldn't change owner/group of %v to (uid: %v, gid: %v): %w
• failed to write vault token to secrets dir: %v
• failed to write vault token: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/a6db707644c565fe. Report an issue: GitHub.