juanfont/headscale · critical

recovering user_id on untagged nodes: %w

Error message

recovering user_id on untagged nodes: %w

What it means

Recovery migration '202606181200-recover-null-tags-node-user-id' fails its UPDATE restoring user_id on untagged nodes from the owning pre-auth key (repairing damage from the buggy first version of the 202602201200 migration that treated tags='null' as tagged). Causes: foreign-key violations when the subquery resolves to a deleted user, lock contention on nodes/pre_auth_keys, or permissions. If the subquery returns NULL the UPDATE writes NULL silently rather than erroring, so real failures are constraint/lock/permission errors.

Source

Thrown at hscontrol/db/db.go:777

				// upgrades are protected by the fixed migration above and find
				// nothing to repair. Recovery is best-effort: the owner is
				// re-derived from the node's pre-auth key, so nodes registered
				// via CLI/OIDC (no pre-auth key) cannot be recovered and must
				// be reassigned manually.
				// Fixes: https://github.com/juanfont/headscale/issues/3323
				ID: "202606181200-recover-null-tags-node-user-id",
				Migrate: func(tx *gorm.DB) error {
					err := tx.Exec(`
UPDATE nodes
SET user_id = (
	SELECT pak.user_id FROM pre_auth_keys pak WHERE pak.id = nodes.auth_key_id
)
WHERE user_id IS NULL
	AND auth_key_id IS NOT NULL
	AND (tags IS NULL OR tags = '' OR tags = '[]' OR tags = 'null');
						`).Error
					if err != nil {
						return fmt.Errorf("recovering user_id on untagged nodes: %w", err)
					}

					return nil
				},
				Rollback: func(db *gorm.DB) error { return nil },
			},
			{
				// Add an optional owning user to API keys so the v2 API can
				// create user-owned (untagged) auth keys, mirroring Tailscale's
				// "key owned by the creating identity".
				ID: "202606191500-api-key-user-id",
				Migrate: func(tx *gorm.DB) error {
					if !tx.Migrator().HasColumn(&types.APIKey{}, "user_id") {
						err := tx.Migrator().AddColumn(&types.APIKey{}, "user_id")
						if err != nil {
							return fmt.Errorf("adding user_id to api_keys: %w", err)
						}
					}

View on GitHub (pinned to 565fd254d0)

Solutions

1. Check the wrapped error: FK violation means the pre-auth key's user was deleted; re-point those keys or pre-assign a valid user, then restart
2. Run with the tailnet quiesced so the correlated-subquery UPDATE is not blocked
3. Verify recovery afterwards: SELECT count(*) FROM nodes WHERE user_id IS NULL AND auth_key_id IS NOT NULL AND (tags IS NULL OR tags IN ('','[]','null')); should be near zero
4. Back up before running - this migration rewrites ownership data derived from pre_auth_keys

// Pre-flight: every untagged node's auth key should map to an existing user
rows, _ := db.Query(`SELECT n.id, pak.user_id FROM nodes n
	JOIN pre_auth_keys pak ON pak.id = n.auth_key_id
	WHERE n.user_id IS NULL AND n.auth_key_id IS NOT NULL
		AND (n.tags IS NULL OR n.tags IN ('', '[]', 'null'))`)
for rows.Next() {
	var nodeID, userID sql.NullInt64
	rows.Scan(&nodeID, &userID)
	if !userID.Valid {
		log.Warnf("node %d's key has no user; recovery will leave user_id NULL", nodeID)
	}
}

When it happens

Trigger: Concurrent sessions locking nodes or pre_auth_keys during the correlated UPDATE; a foreign key on nodes.user_id requiring a valid users row while the matching pre-auth key's user was deleted (FK violation); missing UPDATE privilege.

Common situations: Repairing a 0.29.0 database that already lost user_id on untagged nodes; upgrading while nodes register with the very pre-auth keys being read by the subquery.

Related errors

• automigrating types.PreAuthKey: %w
• updating tags for node %d: %w
• clearing user_id on tagged nodes: %w
• foreign key constraints violated
• version check: %w

AI-assisted analysis of juanfont/headscale@565fd254d0 (2026-08-15). Data as JSON: /api/errors/0a56bcb91f686acd. Report an issue: GitHub.