{"record":{"id":"49272c3a131fd2fd","repo":"hashicorp/nomad","slug":"failed-to-initialize-client-v","errorCode":null,"errorMessage":"failed to initialize client: %v","messagePattern":"failed to initialize client: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"client/client.go","lineNumber":433,"sourceCode":"\t}\n\n\tc.batchNodeUpdates = newBatchNodeUpdates(\n\t\tc.logger,\n\t\tc.updateNodeFromDriver,\n\t\tc.updateNodeFromDevices,\n\t\tc.updateNodeFromCSI,\n\t\tc.updateNodeFromHostVol,\n\t)\n\n\t// Initialize the server manager\n\tc.servers = servers.New(c.logger, c.shutdownCh, c)\n\n\t// Start server manager rebalancing go routine\n\tgo c.servers.Start()\n\n\t// initialize the client\n\tif err := c.init(); err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to initialize client: %v\", err)\n\t}\n\n\t// initialize the dynamic registry (needs to happen after init)\n\tc.dynamicRegistry =\n\t\tdynamicplugins.NewRegistry(c.stateDB, map[string]dynamicplugins.PluginDispenser{\n\t\t\tdynamicplugins.PluginTypeCSIController: func(info *dynamicplugins.PluginInfo) (any, error) {\n\t\t\t\treturn csi.NewClient(info.ConnectionInfo.SocketPath, logger.Named(\"csi_client\").With(\"plugin.name\", info.Name, \"plugin.type\", \"controller\")), nil\n\t\t\t},\n\t\t\tdynamicplugins.PluginTypeCSINode: func(info *dynamicplugins.PluginInfo) (any, error) {\n\t\t\t\treturn csi.NewClient(info.ConnectionInfo.SocketPath, logger.Named(\"csi_client\").With(\"plugin.name\", info.Name, \"plugin.type\", \"client\")), nil\n\t\t\t},\n\t\t})\n\n\t// Setup the clients RPC server\n\tc.setupClientRpc(rpcs)\n\n\t// Initialize the ACL state\n\tc.clientACLResolver.init()","sourceCodeStart":415,"sourceCodeEnd":451,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/client.go#L415-L451","documentation":"NewClient returns this when c.init() fails — the bootstrap phase that creates state/alloc directories, sets up the RPC handler and local state database. The inner error tells which sub-step failed (e.g. state dir, alloc dir, state DB).","triggerScenarios":"Calling client.NewClient with a config whose state_dir/alloc_dir cannot be created (bad path, permissions), whose state database cannot be opened (e.g. corrupt boltdb), or any other init sub-step failure.","commonSituations":"Misconfigured client block paths pointing at non-existent/non-writable mounts; leftover corrupt state_db from a crashed client; running Nomad as a user without access to data_dir; Kubernetes/Docker volumes mounted read-only.","solutions":["Read the wrapped inner error in the message to identify the failing sub-step.","Fix state_dir/alloc_dir paths in the client config: ensure they exist or are creatable and writable by the nomad user.","If the inner error indicates a corrupt state DB, back up and remove the state/alloc directories (Nomad logs the same guidance) — lost allocs will be rescheduled.","Check filesystem mounts are writable and not full before restarting the client agent."],"exampleFix":"// before\nclient {\n  state_dir = \"/mnt/ro-volume/nomad/state\"\n}\n// after\nclient {\n  state_dir = \"/var/lib/nomad/state\"  # writable by nomad user\n}","handlingStrategy":"validation","validationCode":"// before starting the client, validate the data paths\nfor _, d := range []string{cfg.StateDir, cfg.AllocDir} {\n    if d == \"\" { continue }\n    if fi, err := os.Stat(d); err == nil && !fi.IsDir() {\n        return fmt.Errorf(\"%s exists but is not a directory\", d)\n    }\n    if err := os.MkdirAll(d, 0o700); err != nil {\n        return fmt.Errorf(\"cannot create %s: %w\", d, err)\n    }\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always pin state_dir/alloc_dir to explicit, writable paths rather than relying on defaults","Pre-create and chown data directories in provisioning (packer/terraform/systemd)","Monitor disk fullness on data volumes; state DB writes fail when full","Back up state_dir before Nomad upgrades so corrupt-state recovery is possible"],"tags":["client","initialization","filesystem"],"backgroundTag":"client-initialization-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}