juanfont/headscale · error

refreshing NodeStore after IP backfill: %w

Error message

refreshing NodeStore after IP backfill: %w

What it means

Raised by State.BackfillNodeIPs when the IP backfill transaction in the DB succeeded (changes were made) but the follow-up ListNodes read needed to refresh the NodeStore failed. Note the function returns (changes, err) together: the backfill itself committed, only the in-memory refresh is missing. The NodeStore can therefore be stale relative to the database until the next full reload.

Source

Thrown at hscontrol/state/state.go:1076

			return types.NodeView{}, change.Change{}, fmt.Errorf("renaming node: %w", err)
		}
	}

	return s.persistNodeToDB(view)
}

// BackfillNodeIPs assigns IP addresses to nodes that don't have them.
func (s *State) BackfillNodeIPs() ([]string, error) {
	changes, err := s.db.BackfillNodeIPs(s.ipAlloc)
	if err != nil {
		return nil, err
	}

	// Refresh [NodeStore] after IP changes to ensure consistency
	if len(changes) > 0 {
		nodes, err := s.db.ListNodes()
		if err != nil {
			return changes, fmt.Errorf("refreshing NodeStore after IP backfill: %w", err)
		}

		for _, node := range nodes {
			// Preserve online status and NetInfo when refreshing from database
			existingNode, exists := s.nodeStore.GetNode(node.ID)
			if exists && existingNode.Valid() {
				node.IsOnline = new(existingNode.IsOnline().Get())

				// TODO(kradalby): We should ensure we use the same hostinfo and node merge semantics
				// when a node re-registers as we do when it sends a map request (UpdateNodeFromMapRequest).

				// Preserve NetInfo from existing node to prevent loss during backfill
				netInfo := netInfoFromMapRequest(node.ID, existingNode.Hostinfo().AsStruct(), node.Hostinfo)
				node.Hostinfo = existingNode.Hostinfo().AsStruct()
				node.Hostinfo.NetInfo = netInfo
			}
			// TODO(kradalby): This should just update the IP addresses, nothing else in the node store.
			// We should avoid [NodeStore.PutNode] here.

View on GitHub (pinned to 565fd254d0)

Solutions

1. Fix the underlying DB connectivity issue revealed by the wrapped ListNodes error
2. Restart headscale: startup re-runs the backfill idempotently and refreshes the NodeStore from the DB
3. Verify node IPs in the DB match NodeStore afterwards (headscale nodes list) since changes were committed but the in-memory cache was not refreshed

// Verify DB readability before invoking backfill-heavy startup paths:
if _, err := db.ListUsers(); err != nil {
    log.Fatal().Err(err).Msg("database unavailable; fix before backfill")
}

changes, err := s.BackfillNodeIPs()
if err != nil {
    if len(changes) > 0 {
        // Backfill committed; only the store refresh failed. Restart re-runs idempotently.
        log.Error().Err(err).Strs("changed", changes).Msg("backfill committed but NodeStore stale; restarting reconciles")
    }
    return err
}

When it happens

Trigger: Running BackfillNodeIPs (startup migration path for nodes without IPs) where db.BackfillNodeIPs returns a non-empty change set and the subsequent s.db.ListNodes() call errors — DB connection dropped mid-call, query timeout, or transient SQLite lock/Postgres restart.

Common situations: Server restart against a Postgres that is briefly unavailable after the backfill write; SQLite database locked by a concurrent backup; network blip between headscale and the DB during startup.

Related errors

• init state: %w
• database type not supported
• node not found
• configuration error: %w
• version check: %w

AI-assisted analysis of juanfont/headscale@565fd254d0 (2026-08-15). Data as JSON: /api/errors/06ee4ad95bf10561. Report an issue: GitHub.