juanfont/headscale · critical

updating tags for node %d: %w

Error message

updating tags for node %d: %w

What it means

The RequestTags migration fails its raw UPDATE nodes SET tags = ? WHERE id = ? write after merging validated RequestTags. Causes: the tags column missing (preceding rename migration inconsistent), a type constraint on tags in drifted schemas, lock timeout, or - on Postgres - the migration transaction already being aborted by an earlier statement error, after which every subsequent UPDATE fails with 'current transaction is aborted'.

Source

Thrown at hscontrol/db/db.go:686

									Strs("rejected_tags", rejectedTags).
									Msg("RequestTags rejected during migration (not authorized)")
							}

							continue
						}

						mergedTags := append(slices.Clone(existingTags), validatedTags...)
						slices.Sort(mergedTags)
						mergedTags = slices.Compact(mergedTags)

						tagsJSON, err := json.Marshal(mergedTags)
						if err != nil {
							return fmt.Errorf("serializing merged tags for node %d: %w", node.ID, err)
						}

						err = tx.Exec("UPDATE nodes SET tags = ? WHERE id = ?", string(tagsJSON), node.ID).Error
						if err != nil {
							return fmt.Errorf("updating tags for node %d: %w", node.ID, err)
						}

						log.Info().
							EmbedObject(node).
							Strs("validated_tags", validatedTags).
							Strs("rejected_tags", rejectedTags).
							Strs("existing_tags", existingTags).
							Strs("merged_tags", mergedTags).
							Msg("Migrated validated RequestTags from host_info to tags column")
					}

					return nil
				},
				Rollback: func(db *gorm.DB) error { return nil },
			},
			{
				// Clear user_id on tagged nodes.
				// Tagged nodes are owned by their tags, not a user.

View on GitHub (pinned to 565fd254d0)

Solutions

1. Read the wrapped error first: 'no such column: tags' means fix schema/history ordering per the rename migration; 'database is locked' means serialize access; 'transaction is aborted' means find the earlier failing statement in the same log
2. Stop all other headscale instances and admin clients, restart headscale so the migration re-runs atomically
3. Verify host_info JSON for the node ID in the message is well-formed if the error mentions JSON
4. Keep regular backups so a failed migration window can be replayed from a clean snapshot

// Ensure exclusive write access before the upgrade window (Postgres)
rows, _ := db.Query(`SELECT count(*) FROM pg_stat_activity
	WHERE query ILIKE '%nodes%' AND pid <> pg_backend_pid()`)
var writers int
if rows.Next() { rows.Scan(&writers) }
if writers > 0 {
	log.Warnf("%d sessions touching nodes; migration UPDATE may block", writers)
}

// Startup-level retry: transient lock timeouts resolve on a clean second boot
for attempt := 1; attempt <= 3; attempt++ {
	err := runMigrations(db)
	if err == nil { break }
	if !strings.Contains(err.Error(), "locked") || attempt == 3 {
		log.Fatal().Err(err).Msg("migration failed")
	}
	time.Sleep(time.Duration(attempt) * 5 * time.Second)
}

When it happens

Trigger: Writing merged tags for a node while the nodes table is locked by another session, the tags column does not exist due to schema drift from the 202511131445 rename, or a Postgres transaction already aborted by a prior error so this UPDATE fails with 'current transaction is aborted'.

Common situations: Upgrading under peer load with nodes actively checking in; databases restored with mismatched schema; diagnosing only the last error of an aborted transaction instead of the first.

Related errors

• setting auth_key to null on nodes with non-existing keys: %w
• dropping routes table: %w
• renaming table %s to %s_old: %w
• dropping index: %w
• loading nodes for RequestTags migration: %w

AI-assisted analysis of juanfont/headscale@565fd254d0 (2026-08-15). Data as JSON: /api/errors/38bc221320cb553a. Report an issue: GitHub.