juanfont/headscale · critical

listing users: %w

Error message

listing users: %w

What it means

Migration 202505141324 (fixing double slashes in user provider identifiers) failed at ListUsers(tx, nil). The SELECT could not complete: connection failure, lock timeout, or a schema where the users table lacks expected columns for scanning into types.User.

Source

Thrown at hscontrol/db/db.go:225

					// This is a workaround for the fact that the last_seen column
					// was removed in the 202502171819 migration, but only for some
					// beta testers.
					if !tx.Migrator().HasColumn(&types.Node{}, "last_seen") {
						_ = tx.Migrator().AddColumn(&types.Node{}, "last_seen")
					}

					return nil
				},
				Rollback: func(db *gorm.DB) error { return nil },
			},
			// Fix the provider identifier for users that have a double slash in the
			// provider identifier.
			{
				ID: "202505141324",
				Migrate: func(tx *gorm.DB) error {
					users, err := ListUsers(tx, nil)
					if err != nil {
						return fmt.Errorf("listing users: %w", err)
					}

					for _, user := range users {
						user.ProviderIdentifier.String = types.CleanIdentifier(user.ProviderIdentifier.String)

						err := tx.Save(user).Error
						if err != nil {
							return fmt.Errorf("saving user: %w", err)
						}
					}

					return nil
				},
				Rollback: func(db *gorm.DB) error { return nil },
			},
			// v0.27.0
			// Schema migration to ensure all tables match the expected schema.
			// This migration recreates all tables to match the exact structure in schema.sql,

View on GitHub (pinned to 565fd254d0)

Solutions

1. Check the wrapped error for missing column names - that indicates schema drift needing manual repair.
2. For lock/timeout errors, stop concurrent access and retry the upgrade.
3. Back up the database before upgrade so you can retry safely.
4. Confirm headscale binary version matches the database generation path (see version check).

if _, err := db.NewHeadscaleDatabase(cfg); err != nil {
    if strings.Contains(err.Error(), "listing users") {
        // check for missing columns (schema drift) or lock/timeout in the wrapped error
    }
}

When it happens

Trigger: Listing all users during upgrade while another session locks users; Postgres connection dropped; users table drifted from types.User so GORM's generated SELECT references missing columns.

Common situations: OIDC-enabled deployments with many users hitting a slow query timeout; upgrading a database previously run with an incompatible fork.

Related errors

• saving user: %w
• automigrating types.Route: %w
• automigrating types.PreAuthKey: %w
• automigrating types.Node: %w
• adding prefix column: %w

AI-assisted analysis of juanfont/headscale@565fd254d0 (2026-08-15). Data as JSON: /api/errors/7966a1863712c559. Report an issue: GitHub.