juanfont/headscale · critical

dropping index: %w

Error message

dropping index: %w

What it means

Migration '202511122344-remove-newline-index' fails while dropping one of the users-table identity indexes (idx_provider_identifier, idx_name_provider_identifier, idx_name_no_provider_identifier). DROP INDEX IF EXISTS only guards against a missing index; failure indicates missing DROP privilege, lock contention, or (on PostgreSQL) a dependent constraint referencing the index.

Source

Thrown at hscontrol/db/db.go:561

			},
			{
				ID: "202511122344-remove-newline-index",
				Migrate: func(tx *gorm.DB) error {
					// Reformat multi-line indexes to single-line for consistency
					// This migration drops and recreates the three user identity indexes
					// to match the single-line format expected by schema validation

					// Drop existing multi-line indexes
					dropIndexes := []string{
						`DROP INDEX IF EXISTS idx_provider_identifier`,
						`DROP INDEX IF EXISTS idx_name_provider_identifier`,
						`DROP INDEX IF EXISTS idx_name_no_provider_identifier`,
					}

					for _, dropSQL := range dropIndexes {
						err := tx.Exec(dropSQL).Error
						if err != nil {
							return fmt.Errorf("dropping index: %w", err)
						}
					}

					// Recreate indexes in single-line format
					createIndexes := []string{
						`CREATE UNIQUE INDEX idx_provider_identifier ON users(provider_identifier) WHERE provider_identifier IS NOT NULL`,
						`CREATE UNIQUE INDEX idx_name_provider_identifier ON users(name, provider_identifier)`,
						`CREATE UNIQUE INDEX idx_name_no_provider_identifier ON users(name) WHERE provider_identifier IS NULL`,
					}

					for _, createSQL := range createIndexes {
						err := tx.Exec(createSQL).Error
						if err != nil {
							return fmt.Errorf("creating index: %w", err)
						}
					}

					return nil

View on GitHub (pinned to 565fd254d0)

Solutions

1. Read the wrapped error to distinguish 'permission denied' (grant DROP/DDL) from lock timeout (close idle transactions)
2. Restart headscale during a maintenance window when no other clients are connected to the database
3. On PostgreSQL, if the index backs a UNIQUE constraint, drop the constraint instead (ALTER TABLE users DROP CONSTRAINT ...) after taking a backup, then let the migration recreate the index
4. Re-verify after startup that the three indexes exist in single-line form

// Ensure no long-open transactions block DDL before upgrading (Postgres)
rows, _ := db.Query(`SELECT pid, state, xact_start FROM pg_stat_activity
	WHERE state <> 'idle' AND xact_start < now() - interval '1 minute'`)
// terminate or wait; idle-in-transaction sessions cause the DROP to hang or fail

// When wrapping headscale startup: catch, surface the wrapped cause, exit non-zero
if err := db.NewHeadscaleDatabase(cfg); err != nil {
	if strings.Contains(err.Error(), "dropping index:") {
		log.Error().Err(err).Msg("index migration failed; check locks/privileges, then retry startup")
	}
	os.Exit(1)
}

When it happens

Trigger: Startup migration on a database whose users indexes were created multi-line by an older headscale; the DROP fails because another session holds a blocking transaction on users, or the role lacks DROP on the schema, or a UNIQUE constraint depends on the index on PostgreSQL.

Common situations: Upgrading headscale while long-lived admin sessions or pgbouncer pooling keep transactions open on the users table; least-privilege DB roles missing DDL rights; Postgres refusing to drop an index that backs a UNIQUE constraint.

Related errors

• setting auth_key to null on nodes with non-existing keys: %w
• fetching routes: %w
• dropping routes table: %w
• renaming table %s to %s_old: %w
• creating index: %w

AI-assisted analysis of juanfont/headscale@565fd254d0 (2026-08-15). Data as JSON: /api/errors/efe314cadfd66b71. Report an issue: GitHub.