juanfont/headscale · critical

clearing user_id on tagged nodes: %w

Error message

clearing user_id on tagged nodes: %w

What it means

Migration '202602201200-clear-tagged-node-user-id' fails its bulk UPDATE setting user_id = NULL on nodes that carry tags, enforcing the tags-XOR-user ownership rule (fixes issues #3077 and #3323). Failure modes: user_id has a NOT NULL constraint in drifted schemas or a foreign key blocking the NULL write; locks; permissions. On Postgres a NOT NULL user_id makes SET NULL impossible.

Source

Thrown at hscontrol/db/db.go:725

				// tagged, and the ON DELETE CASCADE FK would destroy the
				// tagged nodes if the user were deleted.
				//
				// A nil tags slice marshals to the JSON literal 'null', so
				// untagged nodes can carry tags='null'. That spelling must be
				// excluded alongside '[]' and '' or untagged nodes lose their
				// user. Nodes already detached by the earlier version of this
				// migration are repaired by the recovery migration below.
				// Fixes: https://github.com/juanfont/headscale/issues/3077
				// Fixes: https://github.com/juanfont/headscale/issues/3323
				ID: "202602201200-clear-tagged-node-user-id",
				Migrate: func(tx *gorm.DB) error {
					err := tx.Exec(`
UPDATE nodes
SET user_id = NULL
WHERE tags IS NOT NULL AND tags != '[]' AND tags != '' AND tags != 'null';
						`).Error
					if err != nil {
						return fmt.Errorf("clearing user_id on tagged nodes: %w", err)
					}

					return nil
				},
				Rollback: func(db *gorm.DB) error { return nil },
			},
			{
				// Clear zero-time node expiry values to NULL.
				// Versions before 0.28 persisted a pointer to a zero
				// time.Time as '0001-01-01 00:00:00+00:00' rather than
				// NULL, which 0.29 reports as an expired node. This
				// normalises the existing rows so the column once
				// again means "no expiry" when unset.
				ID: "202605221435-clear-zero-time-node-expiry",
				Migrate: func(tx *gorm.DB) error {
					err := tx.Exec(`
UPDATE nodes
SET expiry = NULL

View on GitHub (pinned to 565fd254d0)

Solutions

1. Check the wrapped error; if NOT NULL constraint: pre-apply ALTER TABLE nodes ALTER COLUMN user_id DROP NOT NULL (Postgres) or rebuild the column nullable (SQLite), then restart so the migration completes
2. Run the upgrade with clients disconnected/quiesced to avoid lock contention on the bulk UPDATE
3. Verify afterwards that tagged nodes have NULL user_id: SELECT count(*) FROM nodes WHERE user_id IS NOT NULL AND tags IS NOT NULL AND tags NOT IN ('[]','','null'); should be 0
4. Back up before upgrading - this migration intentionally severs user ownership of tagged nodes

Example fix

-- before: NOT NULL user_id blocks the migration
ALTER TABLE nodes ALTER COLUMN user_id DROP NOT NULL;
-- after: restart headscale; migration then completes, then verify
SELECT count(*) FROM nodes WHERE user_id IS NOT NULL AND tags IS NOT NULL AND tags != '[]'; -- expect 0

// Pre-flight: user_id must be nullable for SET NULL to succeed
var nullable bool
db.QueryRow(`SELECT is_nullable = 'YES' FROM information_schema.columns
	WHERE table_name = 'nodes' AND column_name = 'user_id'`).Scan(&nullable) // Postgres
if !nullable {
	log.Fatal("nodes.user_id is NOT NULL; run ALTER TABLE nodes ALTER COLUMN user_id DROP NOT NULL before upgrading")
}

When it happens

Trigger: Upgrading a database where nodes.user_id is NOT NULL (older AutoMigrate variants) or where a foreign key constraint rejects NULL; concurrent access holding locks on nodes during the bulk UPDATE.

Common situations: Pre-0.29 databases with user_id NOT NULL on nodes; running the upgrade while tailnet peers are actively polling and writing rows; least-privilege DB roles.

Related errors

• updating tags for node %d: %w
• clearing zero-time node expiry: %w
• recovering user_id on untagged nodes: %w
• foreign key constraints violated
• version check: %w

AI-assisted analysis of juanfont/headscale@565fd254d0 (2026-08-15). Data as JSON: /api/errors/abc8c88f08ca5672. Report an issue: GitHub.