juanfont/headscale · critical

validating schema: %w

Error message

validating schema: %w

What it means

After migrations, headscale validates the sqlite schema digest against the expected schema using squbble (with litestream's _litestream_lock/_litestream_seq tables ignored). A mismatch means the actual table/column layout in the file differs from what this headscale version expects — schema drift between code and database.

Source

Thrown at hscontrol/db/db.go:1042

		sqlConn.SetMaxOpenConns(maxOpenConns)
		defer sqlConn.SetMaxIdleConns(1)
		defer sqlConn.SetMaxOpenConns(1)

		ctx, cancel := context.WithTimeout(context.Background(), contextTimeout)
		defer cancel()

		opts := squibble.DigestOptions{
			IgnoreTables: []string{
				// Litestream tables, these are inserted by
				// litestream and not part of our schema
				// https://litestream.io/how-it-works
				"_litestream_lock",
				"_litestream_seq",
			},
		}

		if err := squibble.Validate(ctx, sqlConn, dbSchema, &opts); err != nil { //nolint:noinlineerr
			return nil, fmt.Errorf("validating schema: %w", err)
		}
	}

	db := HSDatabase{
		DB:  dbConn,
		cfg: cfg,
	}

	return &db, err
}

func openDB(cfg types.DatabaseConfig) (*gorm.DB, error) {
	// TODO(kradalby): Integrate this with zerolog
	var dbLogger logger.Interface
	if cfg.Debug {
		dbLogger = util.NewDBLogWrapper(&log.Logger, cfg.Gorm.SlowThreshold, cfg.Gorm.SkipErrRecordNotFound, cfg.Gorm.ParameterizedQueries)
	} else {
		dbLogger = logger.Default.LogMode(logger.Silent)

View on GitHub (pinned to 565fd254d0)

Solutions

1. Diff the reported digest details against the expected schema to find the offending table/column.
2. If a dev build mutated the schema, dump data, recreate the database with the release binary, and re-import nodes.
3. If this follows a downgrade, upgrade back to the binary version whose migrations produced the current schema.
4. Never let dev builds write to the production database — dev builds skip setDatabaseVersion precisely to avoid this class of drift.

// Pre-flight before pointing a release binary at a DB a dev build touched:
// compare table digests or simply never share DBs between build types.
// Quick structural check:
//   sqlite3 headscale.db '.schema nodes' | head -40
// and diff against the release binary's expected nodes schema.

// Not catchable meaningfully — squbble validation is a hard gate.
// On failure, restore a schema-consistent database (backup/replica) or
// recreate and re-import. Do not bypass by deleting tables.

When it happens

Trigger: Schema was modified outside the migration path: AutoMigrate created extra/altered columns in a prior dev build, a binary downgrade left newer columns in place, or the DB file was hand-edited. Litestream-ignored tables are excluded, so only real headscale tables count.

Common situations: Running a dev build (which skips version stamping and may have run AutoMigrate) and then switching back to a release binary; restoring a replica restored at a different schema version; editing the sqlite file with an external tool.

Related errors

• foreign key constraints violated
• path cannot be empty
• opening database: %w
• saving API key to database: %w
• automigrating types.Route: %w

AI-assisted analysis of juanfont/headscale@565fd254d0 (2026-08-15). Data as JSON: /api/errors/c7bf2b3df4204f43. Report an issue: GitHub.