juanfont/headscale · error

restoring foreign keys: %w

Error message

restoring foreign keys: %w

What it means

After the FK-off early-migration phase on sqlite, headscale re-enables enforcement with PRAGMA foreign_keys = ON before running all remaining migrations. This error means the re-enable statement failed — almost always the same lock/connection problem as error 408, hitting at a slightly later point in startup.

Source

Thrown at hscontrol/db/db.go:1179

	if cfg.Type == types.DatabaseSqlite {
		// SQLite: Run the early migrations that GORM cannot handle safely with
		// foreign keys enabled (route and pre-auth-key automigrations) with FK
		// disabled, then run everything else with FK enabled.
		//
		// NO NEW MIGRATIONS SHOULD RUN WITH FK DISABLED. As of 2025-07-02, all
		// new migrations must run with foreign keys enabled via the
		// migrations.Migrate() call below.
		if err := dbConn.Exec("PRAGMA foreign_keys = OFF").Error; err != nil { //nolint:noinlineerr
			return fmt.Errorf("disabling foreign keys: %w", err)
		}

		// Run up to and including the last migration that requires FK disabled.
		if err := migrations.MigrateTo("202501311657"); err != nil { //nolint:noinlineerr
			return fmt.Errorf("running migration 202501311657: %w", err)
		}

		if err := dbConn.Exec("PRAGMA foreign_keys = ON").Error; err != nil { //nolint:noinlineerr
			return fmt.Errorf("restoring foreign keys: %w", err)
		}

		// Run the rest of the migrations
		if err := migrations.Migrate(); err != nil { //nolint:noinlineerr
			return err
		}

		// Check for constraint violations at the end
		type constraintViolation struct {
			Table           string
			RowID           int
			Parent          string
			ConstraintIndex int
		}

		var violatedConstraints []constraintViolation

		rows, err := dbConn.Raw("PRAGMA foreign_key_check").Rows()

View on GitHub (pinned to 565fd254d0)

Solutions

1. Eliminate concurrent access to the sqlite file (single headscale instance, replication-only litestream).
2. Check disk space and filesystem health on the volume hosting the DB.
3. Restart headscale once the conflicting holder is gone — migrations are idempotent and will resume.

// Same handling as error 408: if lock-related, remove the conflicting
// holder and restart; migrations resume idempotently.

When it happens

Trigger: Connection degraded or the database got locked between the early migrations finishing and the pragma being re-enabled (e.g. litestream grabbing an exclusive lock, disk filling up mid-startup).

Common situations: Same class as error 408: concurrent writers on the sqlite file, hardware/disk issues, or a container whose volume goes away during startup.

Related errors

• disabling foreign keys: %w
• setting auth_key to null on nodes with non-existing keys: %w
• dropping routes table: %w
• renaming table %s to %s_old: %w
• foreign key constraints violated

AI-assisted analysis of juanfont/headscale@565fd254d0 (2026-08-15). Data as JSON: /api/errors/783d4e2cd58692d5. Report an issue: GitHub.