juanfont/headscale · critical

adding prefix column: %w

Error message

adding prefix column: %w

What it means

Thrown by the headscale database migration '202511011637-preauthkey-bcrypt' when GORM's Migrator().AddColumn fails to add the 'prefix' column to the pre_auth_keys table. The underlying error is wrapped (%w), so the real cause (permissions, disk full, lock timeout, corrupted schema) is in the wrapped message. Migrations run at headscale startup inside a transaction, so failure aborts boot.

Source

Thrown at hscontrol/db/db.go:521

			// Any new migrations should be added after the comment below and follow
			// the rules it sets out.

			// From this point, the following rules must be followed:
			// - NEVER use gorm.AutoMigrate, write the exact migration steps needed
			// - AutoMigrate depends on the struct staying exactly the same, which it won't over time.
			// - Never write migrations that requires foreign keys to be disabled.
			// - ALL errors in migrations must be handled properly.

			{
				// Add columns for prefix and hash for pre auth keys, implementing
				// them with the same security model as api keys.
				ID: "202511011637-preauthkey-bcrypt",
				Migrate: func(tx *gorm.DB) error {
					// Check and add prefix column if it doesn't exist
					if !tx.Migrator().HasColumn(&types.PreAuthKey{}, "prefix") {
						err := tx.Migrator().AddColumn(&types.PreAuthKey{}, "prefix")
						if err != nil {
							return fmt.Errorf("adding prefix column: %w", err)
						}
					}

					// Check and add hash column if it doesn't exist
					if !tx.Migrator().HasColumn(&types.PreAuthKey{}, "hash") {
						err := tx.Migrator().AddColumn(&types.PreAuthKey{}, "hash")
						if err != nil {
							return fmt.Errorf("adding hash column: %w", err)
						}
					}

					// Create partial unique index to allow multiple legacy keys (NULL/empty prefix)
					// while enforcing uniqueness for new bcrypt-based keys
					err := tx.Exec("CREATE UNIQUE INDEX IF NOT EXISTS idx_pre_auth_keys_prefix ON pre_auth_keys(prefix) WHERE prefix IS NOT NULL AND prefix != ''").Error
					if err != nil {
						return fmt.Errorf("creating prefix index: %w", err)
					}

View on GitHub (pinned to 565fd254d0)

Solutions

1. Read the wrapped error in the startup log - it names the real DB failure (e.g. 'permission denied', 'database is locked', 'no space left on device') and dictates the fix
2. Ensure only one headscale process is running against the database (SQLite single-writer lock); stop duplicate containers/instances
3. Grant the headscale DB user DDL rights (ALTER TABLE, CREATE INDEX) or have a DBA pre-apply the migration
4. Free disk space / check volume quotas and vacuum a bloated SQLite database
5. Restore from backup if a prior crashed migration left the schema inconsistent, then re-run startup so the transaction replays cleanly

Example fix

// before: DB user without DDL rights
CREATE USER headscale; GRANT SELECT,INSERT,UPDATE,DELETE ON ALL TABLES IN SCHEMA public TO headscale;
// after: include DDL needed by migrations
CREATE USER headscale; GRANT SELECT,INSERT,UPDATE,DELETE,ALTER,CREATE,INDEX,DROP ON ALL TABLES IN SCHEMA public TO headscale;

// Before starting headscale, verify the DB is readable and DDL-writable
import (
	"database/sql"
	"fmt"
)

func checkMigratable(db *sql.DB) error {
	if err := db.Ping(); err != nil {
		return fmt.Errorf("db not reachable: %w", err)
	}
	if _, err := db.Exec("CREATE TABLE IF NOT EXISTS _migration_probe(id integer)"); err != nil {
		return fmt.Errorf("db not DDL-writable (locked/readonly/permissions): %w", err)
	}
	db.Exec("DROP TABLE IF EXISTS _migration_probe")
	return nil
}

When it happens

Trigger: Starting headscale against a database that predates 2025-11-01 and is missing pre_auth_keys.prefix, while the DB user lacks ALTER privileges, the SQLite file is locked by another headscale instance, the disk is full, or a previous interrupted migration left the schema half-updated.

Common situations: Upgrading headscale from a pre-0.26/0.27 database; running two headscale instances against the same SQLite file; DB user created with only DML (not DDL) rights on PostgreSQL; Docker volume out of space; NFS-mounted SQLite database with locking problems.

Related errors

• automigrating types.Route: %w
• clearing expiry on tagged nodes: %w
• automigrating types.Node: %w
• checking if table %s exists: %w
• adding hash column: %w

AI-assisted analysis of juanfont/headscale@565fd254d0 (2026-08-15). Data as JSON: /api/errors/3d0692a8c99c017e. Report an issue: GitHub.