hashicorp/nomad · error

unable to create unix socket for Consul HTTP endpoint: %w

Error message

unable to create unix socket for Consul HTTP endpoint: %w

What it means

The hook creates a Unix domain socket at hostHTTPSockPath so Consul's HTTP API can be reached securely from within the alloc's network namespace. net.Listen("unix", ...) failed — typically because the path already exists and is stale, the directory doesn't exist, or permission/filesystem constraints prevent socket creation. The underlying OS error is wrapped for diagnosis.

Source

Thrown at client/allocrunner/consul_http_sock_hook.go:228

	// consul http dest addr
	destAddr := p.config.Addr
	if destAddr == "" {
		return errors.New("consul address must be set on nomad client")
	}

	socketFile := allocdir.AllocHTTPSocket
	if p.config.Name != structs.ConsulDefaultCluster && p.config.Name != "" {
		socketFile = filepath.Join(allocdir.SharedAllocName, allocdir.TmpDirName,
			"consul_"+p.config.Name+"_http.sock")
	}
	hostHTTPSockPath := filepath.Join(p.allocDir.AllocDirPath(), socketFile)
	if err := maybeRemoveOldSocket(hostHTTPSockPath); err != nil {
		return err
	}

	listener, err := net.Listen("unix", hostHTTPSockPath)
	if err != nil {
		return fmt.Errorf("unable to create unix socket for Consul HTTP endpoint: %w", err)
	}

	// The Consul HTTP socket should be usable by all users in case a task is
	// running as a non-privileged user. Unix does not allow setting domain
	// socket permissions when creating the file, so we must manually call
	// chmod afterwards.
	if err := os.Chmod(hostHTTPSockPath, os.ModePerm); err != nil {
		return fmt.Errorf("unable to set permissions on unix socket: %w", err)
	}

	go func() {
		proxy(p.ctx, p.logger, destAddr, listener)
		p.cancel()
		close(p.doneCh)
	}()

	p.runOnce = true
	return nil

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Remove the stale socket file at hostHTTPSockPath manually (rm <path>) or ensure maybeRemoveOldSocket succeeded, then restart the alloc
2. Check the wrapped OS error: EACCES → fix directory permissions; ENOENT → create the parent directory
3. Move Nomad's client data_dir off filesystems that don't support unix sockets (NFS) to local disk
4. Restart the Nomad client agent to clean up leaked sockets from crashed allocs

Example fix

// before: listen fails on stale socket
listener, err := net.Listen("unix", hostHTTPSockPath)

// after: force-remove then listen
if err := maybeRemoveOldSocket(hostHTTPSockPath); err != nil {
	return err
}
listener, err := net.Listen("unix", hostHTTPSockPath)

sockDir := filepath.Dir(hostHTTPSockPath)
if fi, err := os.Stat(sockDir); err != nil || !fi.IsDir() {
	return fmt.Errorf("socket dir %s missing: %w", sockDir, err)
}
// test unix socket support on this filesystem
testPath := filepath.Join(sockDir, ".socktest")
l, err := net.Listen("unix", testPath)
if err != nil {
	return fmt.Errorf("filesystem %s does not support unix sockets: %w", sockDir, err)
}
l.Close(); os.Remove(testPath)

listener, err := net.Listen("unix", hostHTTPSockPath)
if err != nil {
	if errors.Is(err, syscall.EADDRINUSE) {
		os.Remove(hostHTTPSockPath) // stale socket; retry once
		listener, err = net.Listen("unix", hostHTTPSockPath)
	}
	if err != nil { return err }
}

When it happens

Trigger: run() calls maybeRemoveOldSocket then net.Listen("unix", hostHTTPSockPath); listen fails with EADDRINUSE (stale socket file still present), ENOENT (parent dir missing), EACCES (permissions), or ENOTSUP (path on a filesystem that disallows unix sockets, e.g. some NFS/VFS mounts).

Common situations: Leftover socket from a crashed previous alloc; alloc dir on NFS or tmpfs without socket support; data_dir permissions changed; host path pointing at a read-only mount.

Related errors

• error renewing the lease: %w
• failed to derive Consul token for task %s: %v
• failed to derive Consul token for service %s: %v
• unable to set permissions on unix socket: %w
• unable to remove existing unix socket: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/5791e8829254ca05. Report an issue: GitHub.