{"record":{"id":"0a56bcb91f686acd","repo":"juanfont/headscale","slug":"recovering-user-id-on-untagged-nodes-w","errorCode":null,"errorMessage":"recovering user_id on untagged nodes: %w","messagePattern":"recovering user_id on untagged nodes: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"hscontrol/db/db.go","lineNumber":777,"sourceCode":"\t\t\t\t// upgrades are protected by the fixed migration above and find\n\t\t\t\t// nothing to repair. Recovery is best-effort: the owner is\n\t\t\t\t// re-derived from the node's pre-auth key, so nodes registered\n\t\t\t\t// via CLI/OIDC (no pre-auth key) cannot be recovered and must\n\t\t\t\t// be reassigned manually.\n\t\t\t\t// Fixes: https://github.com/juanfont/headscale/issues/3323\n\t\t\t\tID: \"202606181200-recover-null-tags-node-user-id\",\n\t\t\t\tMigrate: func(tx *gorm.DB) error {\n\t\t\t\t\terr := tx.Exec(`\nUPDATE nodes\nSET user_id = (\n\tSELECT pak.user_id FROM pre_auth_keys pak WHERE pak.id = nodes.auth_key_id\n)\nWHERE user_id IS NULL\n\tAND auth_key_id IS NOT NULL\n\tAND (tags IS NULL OR tags = '' OR tags = '[]' OR tags = 'null');\n\t\t\t\t\t\t`).Error\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\treturn fmt.Errorf(\"recovering user_id on untagged nodes: %w\", err)\n\t\t\t\t\t}\n\n\t\t\t\t\treturn nil\n\t\t\t\t},\n\t\t\t\tRollback: func(db *gorm.DB) error { return nil },\n\t\t\t},\n\t\t\t{\n\t\t\t\t// Add an optional owning user to API keys so the v2 API can\n\t\t\t\t// create user-owned (untagged) auth keys, mirroring Tailscale's\n\t\t\t\t// \"key owned by the creating identity\".\n\t\t\t\tID: \"202606191500-api-key-user-id\",\n\t\t\t\tMigrate: func(tx *gorm.DB) error {\n\t\t\t\t\tif !tx.Migrator().HasColumn(&types.APIKey{}, \"user_id\") {\n\t\t\t\t\t\terr := tx.Migrator().AddColumn(&types.APIKey{}, \"user_id\")\n\t\t\t\t\t\tif err != nil {\n\t\t\t\t\t\t\treturn fmt.Errorf(\"adding user_id to api_keys: %w\", err)\n\t\t\t\t\t\t}\n\t\t\t\t\t}","sourceCodeStart":759,"sourceCodeEnd":795,"githubUrl":"https://github.com/juanfont/headscale/blob/565fd254d06c4c7f9a8cad1714a43445c79ba420/hscontrol/db/db.go#L759-L795","documentation":"Recovery migration '202606181200-recover-null-tags-node-user-id' fails its UPDATE restoring user_id on untagged nodes from the owning pre-auth key (repairing damage from the buggy first version of the 202602201200 migration that treated tags='null' as tagged). Causes: foreign-key violations when the subquery resolves to a deleted user, lock contention on nodes/pre_auth_keys, or permissions. If the subquery returns NULL the UPDATE writes NULL silently rather than erroring, so real failures are constraint/lock/permission errors.","triggerScenarios":"Concurrent sessions locking nodes or pre_auth_keys during the correlated UPDATE; a foreign key on nodes.user_id requiring a valid users row while the matching pre-auth key's user was deleted (FK violation); missing UPDATE privilege.","commonSituations":"Repairing a 0.29.0 database that already lost user_id on untagged nodes; upgrading while nodes register with the very pre-auth keys being read by the subquery.","solutions":["Check the wrapped error: FK violation means the pre-auth key's user was deleted; re-point those keys or pre-assign a valid user, then restart","Run with the tailnet quiesced so the correlated-subquery UPDATE is not blocked","Verify recovery afterwards: SELECT count(*) FROM nodes WHERE user_id IS NULL AND auth_key_id IS NOT NULL AND (tags IS NULL OR tags IN ('','[]','null')); should be near zero","Back up before running - this migration rewrites ownership data derived from pre_auth_keys"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Pre-flight: every untagged node's auth key should map to an existing user\nrows, _ := db.Query(`SELECT n.id, pak.user_id FROM nodes n\n\tJOIN pre_auth_keys pak ON pak.id = n.auth_key_id\n\tWHERE n.user_id IS NULL AND n.auth_key_id IS NOT NULL\n\t\tAND (n.tags IS NULL OR n.tags IN ('', '[]', 'null'))`)\nfor rows.Next() {\n\tvar nodeID, userID sql.NullInt64\n\trows.Scan(&nodeID, &userID)\n\tif !userID.Valid {\n\t\tlog.Warnf(\"node %d's key has no user; recovery will leave user_id NULL\", nodeID)\n\t}\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["If you hit the original 202602201200 bug (untagged nodes lost user_id), back up before the upgrade that carries this recovery","Keep pre_auth_keys.user_id accurate; delete keys together with their users","Quiesce registrations during the repair upgrade"],"tags":["database","migration","data-repair","foreign-key","tags"],"backgroundTag":null,"analyzedSha":"565fd254d06c4c7f9a8cad1714a43445c79ba420","analyzedAt":"2026-08-15T13:12:30.133Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}