{"record":{"id":"80495631ed6329f4","repo":"hashicorp/nomad","slug":"error-saving-client-identity-w","errorCode":null,"errorMessage":"error saving client identity: %w","messagePattern":"error saving client identity: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/client.go","lineNumber":2329,"sourceCode":"\tc.EnterpriseClient.SetFeatures(resp.Features)\n\treturn nil\n}\n\nfunc (c *Client) handleNodeUpdateResponse(resp structs.NodeUpdateResponse) error {\n\t// Update the number of nodes in the cluster so we can adjust our server\n\t// rebalance rate.\n\tc.servers.SetNumNodes(resp.NumNodes)\n\n\t// If the response includes a new identity, set it and save it to the state\n\t// DB.\n\t//\n\t// In the unlikely event that we cannot write the identity to the state DB,\n\t// we do not want to set the client identity token. That would mean the\n\t// client memory state and persistent state DB are out of sync. Instead, we\n\t// return an error and wait until the next heartbeat to try again.\n\tif resp.SignedIdentity != nil {\n\t\tif err := c.stateDB.PutNodeIdentity(*resp.SignedIdentity); err != nil {\n\t\t\treturn fmt.Errorf(\"error saving client identity: %w\", err)\n\t\t}\n\t\tc.setNodeIdentityToken(*resp.SignedIdentity)\n\n\t\t// If the operator forced this renewal, reset the flag so that we don't\n\t\t// keep renewing the identity on every heartbeat.\n\t\tc.identityForceRenewal.Store(false)\n\t}\n\n\t// Convert []*NodeServerInfo to []*servers.Server\n\tnomadServers := make([]*servers.Server, 0, len(resp.Servers))\n\tfor _, s := range resp.Servers {\n\t\taddr, err := resolveServer(s.RPCAdvertiseAddr)\n\t\tif err != nil {\n\t\t\tc.logger.Warn(\"ignoring invalid server\", \"error\", err, \"server\", s.RPCAdvertiseAddr)\n\t\t\tcontinue\n\t\t}\n\t\te := &servers.Server{Addr: addr}\n\t\tnomadServers = append(nomadServers, e)","sourceCodeStart":2311,"sourceCodeEnd":2347,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/client.go#L2311-L2347","documentation":"When the server returns a renewed SignedIdentity in the heartbeat response, the client must persist it to the state DB before setting it in memory. If stateDB.PutNodeIdentity fails, the identity is intentionally NOT applied in memory (to avoid memory/disk divergence) and this error is returned; the client retries on the next heartbeat.","triggerScenarios":"resp.SignedIdentity is non-nil but c.stateDB.PutNodeIdentity fails — corrupt/locked bolt DB, disk full, read-only filesystem, or I/O error on data_dir during a heartbeat with identity (re)issuance.","commonSituations":"Disk full on the client node; state.db corrupted after a crash; permissions on data_dir changed (e.g. by config management); first identity issuance right after client bootstrap on a node with failing storage.","solutions":["Read the wrapped error to identify the storage cause (enospc, permission denied, bolt corruption)","Free disk space or fix data_dir permissions so the state DB is writable","Restart the client; identity renewal is retried on the next heartbeat so the error can self-heal once storage is fixed","If state.db is corrupt, stop Nomad, back up and remove it, then restart (node re-registers)"],"exampleFix":"# before\n$ df -h /var/lib/nomad\n/dev/sda1  100%  used\n// after\nsudo journalctl --vacuum-size=100M\nsudo systemctl restart nomad","handlingStrategy":"retry","validationCode":"// ensure state DB is writable before heartbeats expect identity writes\nif err := probeWrite(cfg.DataDir); err != nil {\n\treturn fmt.Errorf(\"identity persistence will fail: data_dir not writable: %w\", err)\n}\nif freeDisk(cfg.DataDir) < minFreeBytes {\n\treturn fmt.Errorf(\"insufficient disk for identity persistence\")\n}","typeGuard":null,"tryCatchPattern":"if err := stateDB.PutNodeIdentity(identity); err != nil {\n\tlogger.Error(\"identity not persisted; memory token NOT applied\", \"err\", err)\n\t// do not set in-memory token; retry on next heartbeat\n\treturn retryOnNextHeartbeat()\n}","preventionTips":["Alert on disk-full conditions on client nodes","Keep data_dir permissions managed and consistent","Restart the client after fixing storage — renewal retries automatically","Back up state.db before maintenance to avoid corruption loss"],"tags":["nomad-client","state-db","identity"],"backgroundTag":"identity-persistence-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}