juanfont/headscale · critical

fetching routes: %w

Error message

fetching routes: %w

What it means

Migration 202502131714 failed reading all rows from the legacy routes table (tx.Find(&routes)) before converting enabled routes into per-node approved route lists. This is a SELECT failure: connection dropped, lock timeout, or rows that cannot scan into types.Route (e.g. malformed prefix data written by a fork).

Source

Thrown at hscontrol/db/db.go:164

			// Migrate all routes from the Route table to the new field ApprovedRoutes
			// in the Node table. Then drop the Route table.
			{
				ID: "202502131714",
				Migrate: func(tx *gorm.DB) error {
					if !tx.Migrator().HasColumn(&types.Node{}, "approved_routes") {
						err := tx.Migrator().AddColumn(&types.Node{}, "approved_routes")
						if err != nil {
							return fmt.Errorf("adding column types.Node: %w", err)
						}
					}

					nodeRoutes := map[uint64][]netip.Prefix{}

					var routes []types.Route //nolint:staticcheck // SA1019: Route kept for migrations

					err = tx.Find(&routes).Error
					if err != nil {
						return fmt.Errorf("fetching routes: %w", err)
					}

					for _, route := range routes {
						if route.Enabled {
							nodeRoutes[route.NodeID] = append(nodeRoutes[route.NodeID], route.Prefix)
						}
					}

					for nodeID, routes := range nodeRoutes {
						slices.SortFunc(routes, netip.Prefix.Compare)
						routes = slices.Compact(routes)

						data, _ := json.Marshal(routes)

						err = tx.Model(&types.Node{}).Where("id = ?", nodeID).Update("approved_routes", data).Error
						if err != nil {
							return fmt.Errorf("saving approved routes to new column: %w", err)
						}

View on GitHub (pinned to 565fd254d0)

Solutions

1. Inspect the wrapped error - a scan/parse error names the offending row; a timeout suggests lock contention.
2. Increase the statement timeout / retry during a quiet window.
3. Back up, then manually inspect the routes table for malformed prefix values and fix or delete them.
4. Retry startup - the migration runs in a transaction, so no partial state persists.

if _, err := db.NewHeadscaleDatabase(cfg); err != nil {
    if strings.Contains(err.Error(), "fetching routes") {
        // SELECT failed: inspect wrapped error for scan errors (bad prefix data) vs timeouts
    }
}

When it happens

Trigger: Query timeout on a large routes table; Postgres connection reset mid-migration; a prefix column value that netip.Prefix cannot parse when scanned.

Common situations: Upgrading databases with many thousands of routes; network blips between headscale and Postgres; databases previously touched by modified builds that wrote invalid prefix strings.

Related errors

• adding column types.Node: %w
• saving approved routes to new column: %w
• adding prefix column: %w
• dropping index: %w
• creating oauth_clients index: %w

AI-assisted analysis of juanfont/headscale@565fd254d0 (2026-08-15). Data as JSON: /api/errors/ec0a523f8865e4ef. Report an issue: GitHub.