juanfont/headscale · critical

setting auth_key to null on nodes with non-existing keys: %w

Error message

setting auth_key to null on nodes with non-existing keys: %w

What it means

Migration 202502070949 failed while nulling auth_key_id on nodes that reference non-existent pre-auth keys. The raw UPDATE ... WHERE auth_key_id NOT IN (SELECT id FROM pre_auth_keys) only fails on infrastructure problems - the code already guards with HasTable, so the cause is a locked, read-only, or permission-restricted database.

Source

Thrown at hscontrol/db/db.go:137

					return nil
				},
				Rollback: func(db *gorm.DB) error { return nil },
			},
			// Ensure there are no nodes referring to a deleted preauthkey.
			{
				ID: "202502070949",
				Migrate: func(tx *gorm.DB) error {
					if tx.Migrator().HasTable(&types.PreAuthKey{}) {
						err := tx.Exec(`
UPDATE nodes
SET auth_key_id = NULL
WHERE auth_key_id IS NOT NULL
AND auth_key_id NOT IN (
    SELECT id FROM pre_auth_keys
);
							`).Error
						if err != nil {
							return fmt.Errorf("setting auth_key to null on nodes with non-existing keys: %w", err)
						}
					}

					return nil
				},
				Rollback: func(db *gorm.DB) error { return nil },
			},
			// v0.26.0
			// Migrate all routes from the Route table to the new field ApprovedRoutes
			// in the Node table. Then drop the Route table.
			{
				ID: "202502131714",
				Migrate: func(tx *gorm.DB) error {
					if !tx.Migrator().HasColumn(&types.Node{}, "approved_routes") {
						err := tx.Migrator().AddColumn(&types.Node{}, "approved_routes")
						if err != nil {
							return fmt.Errorf("adding column types.Node: %w", err)
						}

View on GitHub (pinned to 565fd254d0)

Solutions

1. Ensure only one headscale process runs during startup; stop CLI usage until migrations finish.
2. Check write permissions on the SQLite file AND its directory (WAL/journal files).
3. For Postgres, grant UPDATE on the nodes table to the migration role.
4. Retry startup after clearing the lock - the migration is idempotent within its transaction.

if _, err := db.NewHeadscaleDatabase(cfg); err != nil {
    if strings.Contains(err.Error(), "setting auth_key to null") {
        // write lock contention: stop concurrent writers and restart once
    }
}

When it happens

Trigger: The UPDATE cannot acquire a write lock (SQLite 'database is locked', Postgres row locks held by another transaction), the database user lacks UPDATE on nodes, or the SQLite file/directory is read-only.

Common situations: Concurrent headscale CLI command running during server startup migration; database on a read-only mount; NFS locking issues with SQLite.

Related errors

• dropping routes table: %w
• renaming table %s to %s_old: %w
• foreign key constraints violated
• automigrating types.Route: %w
• automigrating types.Node: %w

AI-assisted analysis of juanfont/headscale@565fd254d0 (2026-08-15). Data as JSON: /api/errors/726fb7673a24c7e7. Report an issue: GitHub.