{"record":{"id":"a6db707644c565fe","repo":"hashicorp/nomad","slug":"unable-to-remove-existing-unix-socket-w","errorCode":null,"errorMessage":"unable to remove existing unix socket: %w","messagePattern":"unable to remove existing unix socket: %w","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/allocrunner/consul_http_sock_hook.go","lineNumber":270,"sourceCode":"\t// if proxy was never run, no need to wait before shutdown\n\tif !p.runOnce {\n\t\treturn nil\n\t}\n\n\tselect {\n\tcase <-p.doneCh:\n\tcase <-time.After(socketProxyStopWaitTime):\n\t\treturn errSocketProxyTimeout\n\t}\n\n\treturn nil\n}\n\nfunc maybeRemoveOldSocket(socketPath string) error {\n\t_, err := os.Stat(socketPath)\n\tif err == nil {\n\t\tif err = os.Remove(socketPath); err != nil {\n\t\t\treturn fmt.Errorf(\"unable to remove existing unix socket: %w\", err)\n\t\t}\n\t}\n\treturn nil\n}\n","sourceCodeStart":252,"sourceCodeEnd":275,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocrunner/consul_http_sock_hook.go#L252-L275","documentation":"Nomad's Consul HTTP socket hook creates a unix socket in the alloc dir so tasks can reach Consul via a proxied socket. maybeRemoveOldSocket stats the socket path and removes a stale socket left from a previous allocation run. This error wraps the os.Remove failure when the file exists but cannot be deleted, typically due to filesystem permissions or a busy mount.","triggerScenarios":"os.Stat(socketPath) succeeds (file/socket exists) and os.Remove(socketPath) fails with e.g. EACCES, EPERM, or EBUSY during alloc runner hook execution.","commonSituations":"Alloc data dir owned by a different user after a Nomad upgrade or user change; read-only or full host volume backing the alloc dir; the socket path is a mount point that cannot be unlinked; leftover sockets after an unclean node shutdown on bind-mounted volumes.","solutions":["Check ownership/permissions of the alloc dir socket path and fix them (chown to the nomad user) or remove the stale file manually","Ensure the parent directory is writable by the Nomad agent user and not mounted read-only","Restart the Nomad client or stop the task so the socket is not in use, then retry the allocation","As a last resort, clean the affected alloc dir (nomad system reap / remove client data dir for that alloc)"],"exampleFix":"// before\n# ls -l <alloc_dir>/consul_http.sock  -> owned by root, nomad can't unlink\n// after\n# chown nomad:nomad <alloc_dir> && rm -f <alloc_dir>/consul_http.sock\n# nomad alloc stop <alloc_id>  # reschedules and recreates socket","handlingStrategy":"validation","validationCode":"const st = await fs.promises.stat(socketPath).catch(e => e.code === 'ENOENT' ? null : Promise.reject(e));\nif (st) await fs.promises.access(path.dirname(socketPath), fs.constants.W_OK | fs.constants.X_OK);","typeGuard":null,"tryCatchPattern":"try {\n  await fs.promises.rm(socketPath, { force: true });\n} catch (err) {\n  if (err.code === 'EACCES' || err.code === 'EPERM' || err.code === 'EBUSY') {\n    // surface actionable hint: check alloc dir ownership / mount state\n    throw new Error(`cannot remove stale socket ${socketPath}: ${err.message} (check perms/mounts)`);\n  }\n  throw err;\n}","preventionTips":["Ensure the alloc data dir is consistently owned by the Nomad user across upgrades","Never mount the alloc dir read-only","After unclean shutdowns, clean stale sockets before restarting allocs","Monitor for EACCES/EBUSY on the client's data_dir volume"],"tags":["filesystem","unix-socket","permissions","nomad"],"backgroundTag":"unix-socket-remove-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}