juanfont/headscale · critical

clearing zero-time node expiry: %w

Error message

clearing zero-time node expiry: %w

What it means

Migration '202605221435-clear-zero-time-node-expiry' fails its UPDATE setting expiry = NULL where expiry predates 1900, normalising '0001-01-01' zero-time values that 0.29 misreports as expired. Causes: expiry column having a NOT NULL constraint in drifted schemas, lock contention, or permissions - same class as the other bulk UPDATE migrations.

Source

Thrown at hscontrol/db/db.go:747

				},
				Rollback: func(db *gorm.DB) error { return nil },
			},
			{
				// Clear zero-time node expiry values to NULL.
				// Versions before 0.28 persisted a pointer to a zero
				// time.Time as '0001-01-01 00:00:00+00:00' rather than
				// NULL, which 0.29 reports as an expired node. This
				// normalises the existing rows so the column once
				// again means "no expiry" when unset.
				ID: "202605221435-clear-zero-time-node-expiry",
				Migrate: func(tx *gorm.DB) error {
					err := tx.Exec(`
UPDATE nodes
SET expiry = NULL
WHERE expiry IS NOT NULL AND expiry < '1900-01-01';
						`).Error
					if err != nil {
						return fmt.Errorf("clearing zero-time node expiry: %w", err)
					}

					return nil
				},
				Rollback: func(db *gorm.DB) error { return nil },
			},
			{
				// Recover user_id on untagged nodes detached by the earlier
				// version of 202602201200-clear-tagged-node-user-id, which
				// treated tags='null' as tagged and cleared the user. This
				// repairs databases that already upgraded to 0.29.0; fresh
				// upgrades are protected by the fixed migration above and find
				// nothing to repair. Recovery is best-effort: the owner is
				// re-derived from the node's pre-auth key, so nodes registered
				// via CLI/OIDC (no pre-auth key) cannot be recovered and must
				// be reassigned manually.
				// Fixes: https://github.com/juanfont/headscale/issues/3323
				ID: "202606181200-recover-null-tags-node-user-id",

View on GitHub (pinned to 565fd254d0)

Solutions

1. From the wrapped error: if NOT NULL violation, pre-apply ALTER TABLE nodes ALTER COLUMN expiry DROP NOT NULL, then restart headscale
2. Upgrade during a maintenance window to avoid lock timeouts on nodes
3. Post-migration sanity check: SELECT count(*) FROM nodes WHERE expiry IS NOT NULL AND expiry < '1900-01-01'; should be 0
4. Take a backup; the NULL rewrite changes semantics (NULL = no expiry) so verify expired-node reporting afterwards

Example fix

-- before: NOT NULL expiry blocks the NULL write
ALTER TABLE nodes ALTER COLUMN expiry DROP NOT NULL;
-- after: restart headscale, then confirm zero-time rows are gone
SELECT count(*) FROM nodes WHERE expiry < '1900-01-01'; -- expect 0

// Pre-flight: expiry must be nullable
var nullable bool
db.QueryRow(`SELECT is_nullable = 'YES' FROM information_schema.columns
	WHERE table_name = 'nodes' AND column_name = 'expiry'`).Scan(&nullable)
if !nullable {
	log.Fatal("nodes.expiry is NOT NULL; make it nullable before upgrading")
}
// Optional: preview affected rows
// SELECT count(*) FROM nodes WHERE expiry IS NOT NULL AND expiry < '1900-01-01';

When it happens

Trigger: Schema drift leaving nodes.expiry NOT NULL while the migration sets NULL; long-running transactions blocking the range UPDATE; missing UPDATE privilege.

Common situations: Databases created by very old headscale versions where expiry was NOT NULL; upgrading under load.

Related errors

• clearing user_id on tagged nodes: %w
• foreign key constraints violated
• version check: %w
• automigrating types.Route: %w
• automigrating types.PreAuthKey: %w

AI-assisted analysis of juanfont/headscale@565fd254d0 (2026-08-15). Data as JSON: /api/errors/5655747c1e51ddba. Report an issue: GitHub.