juanfont/headscale · critical

clearing expiry on tagged nodes: %w

Error message

clearing expiry on tagged nodes: %w

What it means

Thrown inside GORM migration '202607241200-clear-tagged-node-expiry', which nulls out the expiry column on tagged nodes (tagged nodes cannot expire, per headscale's tags-XOR-user-ownership model). The wrapping error means the raw UPDATE statement failed at the SQL level. The underlying driver error is chained via %w and tells the real story.

Source

Thrown at hscontrol/db/db.go:923

				// owned by its tags and never expires (KB 1068), but a buggy
				// handleLogout stamped a past expiry on it, leaving it
				// permanently Expired and unable to re-authenticate. The
				// buggy writer is fixed, so this only repairs rows written
				// before the upgrade; a fixed server cannot recreate them.
				// Match the tagged-node predicate the earlier
				// clear-tagged-node-user-id migration uses (a nil tags slice
				// marshals to 'null', so exclude it).
				// Fixes: https://github.com/juanfont/headscale/issues/3371
				ID: "202607241200-clear-tagged-node-expiry",
				Migrate: func(tx *gorm.DB) error {
					err := tx.Exec(`
UPDATE nodes
SET expiry = NULL
WHERE tags IS NOT NULL AND tags != '[]' AND tags != '' AND tags != 'null'
	AND expiry IS NOT NULL;
						`).Error
					if err != nil {
						return fmt.Errorf("clearing expiry on tagged nodes: %w", err)
					}

					return nil
				},
				Rollback: func(db *gorm.DB) error { return nil },
			},
		},
	)

	migrations.InitSchema(func(tx *gorm.DB) error {
		// Create all tables using AutoMigrate
		err := tx.AutoMigrate(
			&types.User{},
			&types.PreAuthKey{},
			&types.APIKey{},
			&types.Node{},
			&types.Policy{},
			&types.OAuthClient{},

View on GitHub (pinned to 565fd254d0)

Solutions

1. Check the chained driver error in the same log line — it names the exact SQL failure (locked, no such table, disk I/O).
2. Ensure only one headscale process is running against the sqlite path; stop litestream or take a backup before migrating.
3. Verify filesystem permissions and free space on the volume holding cfg.Database.Sqlite.Path.
4. If the nodes table is missing/corrupt, restore from backup (or litestream replica) and re-run startup so the migration retries.
5. Confirm the database was not created by a much older/newer headscale with an incompatible nodes schema; migrate stepwise through intermediate versions.

// Before startup: ensure the sqlite file is not held by another process
import "os"

func canLock(path string) bool {
    f, err := os.OpenFile(path+".probe", os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o644)
    if err != nil { return false }
    f.Close()
    os.Remove(path + ".probe")
    return true
}

// headscale surfaces this at startup and exits; handle at the orchestrator level:
// systemd unit with Restart=on-failure and a backoff gives lock-transient
// migration failures a chance to succeed on retry. Inspect the wrapped %w
// error in logs to distinguish 'database is locked' (retry helps) from
// SQL/logic errors (retry will not help).

When it happens

Trigger: Running headscale startup (or an explicit migrate) against a database where the UPDATE nodes SET expiry = NULL ... statement cannot execute: table 'nodes' missing/recreated, sqlite database file locked by another process, postgres connection dropped mid-migration, or the tags column has an unexpected type making the comparison invalid.

Common situations: Upgrading headscale to a version containing this migration while another headscale/litestream instance holds the sqlite file; a partially-restored or hand-edited database where the nodes table was dropped; running migrations on a read-only filesystem or full disk.

Related errors

• adding prefix column: %w
• automigrating types.Route: %w
• checking if table %s exists: %w
• foreign key constraints violated
• opening database: %w

AI-assisted analysis of juanfont/headscale@565fd254d0 (2026-08-15). Data as JSON: /api/errors/b79844c3bcd65747. Report an issue: GitHub.