{"record":{"id":"06ee4ad95bf10561","repo":"juanfont/headscale","slug":"refreshing-nodestore-after-ip-backfill-w","errorCode":null,"errorMessage":"refreshing NodeStore after IP backfill: %w","messagePattern":"refreshing NodeStore after IP backfill: %w","errorType":"http","errorClass":null,"httpStatus":500,"severity":"error","filePath":"hscontrol/state/state.go","lineNumber":1076,"sourceCode":"\t\t\treturn types.NodeView{}, change.Change{}, fmt.Errorf(\"renaming node: %w\", err)\n\t\t}\n\t}\n\n\treturn s.persistNodeToDB(view)\n}\n\n// BackfillNodeIPs assigns IP addresses to nodes that don't have them.\nfunc (s *State) BackfillNodeIPs() ([]string, error) {\n\tchanges, err := s.db.BackfillNodeIPs(s.ipAlloc)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\t// Refresh [NodeStore] after IP changes to ensure consistency\n\tif len(changes) > 0 {\n\t\tnodes, err := s.db.ListNodes()\n\t\tif err != nil {\n\t\t\treturn changes, fmt.Errorf(\"refreshing NodeStore after IP backfill: %w\", err)\n\t\t}\n\n\t\tfor _, node := range nodes {\n\t\t\t// Preserve online status and NetInfo when refreshing from database\n\t\t\texistingNode, exists := s.nodeStore.GetNode(node.ID)\n\t\t\tif exists && existingNode.Valid() {\n\t\t\t\tnode.IsOnline = new(existingNode.IsOnline().Get())\n\n\t\t\t\t// TODO(kradalby): We should ensure we use the same hostinfo and node merge semantics\n\t\t\t\t// when a node re-registers as we do when it sends a map request (UpdateNodeFromMapRequest).\n\n\t\t\t\t// Preserve NetInfo from existing node to prevent loss during backfill\n\t\t\t\tnetInfo := netInfoFromMapRequest(node.ID, existingNode.Hostinfo().AsStruct(), node.Hostinfo)\n\t\t\t\tnode.Hostinfo = existingNode.Hostinfo().AsStruct()\n\t\t\t\tnode.Hostinfo.NetInfo = netInfo\n\t\t\t}\n\t\t\t// TODO(kradalby): This should just update the IP addresses, nothing else in the node store.\n\t\t\t// We should avoid [NodeStore.PutNode] here.","sourceCodeStart":1058,"sourceCodeEnd":1094,"githubUrl":"https://github.com/juanfont/headscale/blob/565fd254d06c4c7f9a8cad1714a43445c79ba420/hscontrol/state/state.go#L1058-L1094","documentation":"Raised by State.BackfillNodeIPs when the IP backfill transaction in the DB succeeded (changes were made) but the follow-up ListNodes read needed to refresh the NodeStore failed. Note the function returns (changes, err) together: the backfill itself committed, only the in-memory refresh is missing. The NodeStore can therefore be stale relative to the database until the next full reload.","triggerScenarios":"Running BackfillNodeIPs (startup migration path for nodes without IPs) where db.BackfillNodeIPs returns a non-empty change set and the subsequent s.db.ListNodes() call errors — DB connection dropped mid-call, query timeout, or transient SQLite lock/Postgres restart.","commonSituations":"Server restart against a Postgres that is briefly unavailable after the backfill write; SQLite database locked by a concurrent backup; network blip between headscale and the DB during startup.","solutions":["Fix the underlying DB connectivity issue revealed by the wrapped ListNodes error","Restart headscale: startup re-runs the backfill idempotently and refreshes the NodeStore from the DB","Verify node IPs in the DB match NodeStore afterwards (headscale nodes list) since changes were committed but the in-memory cache was not refreshed"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Verify DB readability before invoking backfill-heavy startup paths:\nif _, err := db.ListUsers(); err != nil {\n    log.Fatal().Err(err).Msg(\"database unavailable; fix before backfill\")\n}","typeGuard":null,"tryCatchPattern":"changes, err := s.BackfillNodeIPs()\nif err != nil {\n    if len(changes) > 0 {\n        // Backfill committed; only the store refresh failed. Restart re-runs idempotently.\n        log.Error().Err(err).Strs(\"changed\", changes).Msg(\"backfill committed but NodeStore stale; restarting reconciles\")\n    }\n    return err\n}","preventionTips":["Treat (changes != nil, err != nil) as 'DB ahead of NodeStore' — plan a restart to reconcile","BackfillNodeIPs is idempotent: safe to re-run after DB connectivity is restored","Monitor DB health before scheduled restarts so startup never hits a half-available database"],"tags":["state","database","backfill","startup","go"],"backgroundTag":null,"analyzedSha":"565fd254d06c4c7f9a8cad1714a43445c79ba420","analyzedAt":"2026-08-15T13:12:30.133Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}