{"record":{"id":"8d7096e9f15fbcc7","repo":"juanfont/headscale","slug":"listing-nodes-to-backfill-ips-w","errorCode":null,"errorMessage":"listing nodes to backfill IPs: %w","messagePattern":"listing nodes to backfill IPs: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"hscontrol/db/ip.go","lineNumber":321,"sourceCode":"// it will be added.\n// If a prefix type has been removed (IPv4 or IPv6), it\n// will remove the IPs in that family from the node.\nfunc (db *HSDatabase) BackfillNodeIPs(i *IPAllocator) ([]string, error) {\n\tvar (\n\t\terr error\n\t\tret []string\n\t)\n\n\terr = db.Write(func(tx *gorm.DB) error {\n\t\tif i == nil {\n\t\t\treturn fmt.Errorf(\"backfilling IPs: %w\", errIPAllocatorNil)\n\t\t}\n\n\t\tlog.Trace().Caller().Msgf(\"starting to backfill IPs\")\n\n\t\tnodes, err := ListNodes(tx)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"listing nodes to backfill IPs: %w\", err)\n\t\t}\n\n\t\tfor _, node := range nodes {\n\t\t\tlog.Trace().Caller().EmbedObject(node).Msg(\"ip backfill check started because node found in database\")\n\n\t\t\tchanged := false\n\t\t\t// IPv4 prefix is set, but node ip is missing, alloc\n\t\t\tif i.prefix4 != nil && node.IPv4 == nil {\n\t\t\t\tret4, err := i.allocateNext(&i.prev4, i.prefix4)\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"allocating IPv4 for node(%d): %w\", node.ID, err)\n\t\t\t\t}\n\n\t\t\t\tnode.IPv4 = ret4\n\t\t\t\tchanged = true\n\n\t\t\t\tret = append(ret, fmt.Sprintf(\"assigned IPv4 %q to Node(%d) %q\", ret4.String(), node.ID, node.Hostname))\n\t\t\t}","sourceCodeStart":303,"sourceCodeEnd":339,"githubUrl":"https://github.com/juanfont/headscale/blob/565fd254d06c4c7f9a8cad1714a43445c79ba420/hscontrol/db/ip.go#L303-L339","documentation":"Inside the backfill write transaction, ListNodes loads every node to decide which need IPs added or removed. This error means the SELECT itself failed at the database layer; the message only adds context, the underlying error (lock timeout, lost connection, corrupt table) is in %w. Because it runs inside db.Write, the transaction rolls back cleanly.","triggerScenarios":"SQLite 'database is locked' from a concurrent writer; PostgreSQL connection dropped mid-transaction; nodes table corruption; a migration that has not run leaving the table in an unexpected shape.","commonSituations":"Running another headscale instance against the same SQLite file; disk-full on the database volume; a crash mid-upgrade leaving a stale -wal/-shm file; Postgres restarted under load.","solutions":["Check the wrapped error with errors.Unwrap to identify lock vs connectivity vs corruption","For SQLite locking, ensure only one headscale process uses the file and busy_timeout is configured","Verify disk space and database integrity (sqlite3 PRAGMA integrity_check / pg connectivity)","Re-run startup once the underlying database issue is fixed; backfill is idempotent"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Verify DB reachability before startup backfill\nif err := db.Ping(); err != nil {\n\treturn fmt.Errorf(\"database unreachable before backfill: %w\", err)\n}","typeGuard":"func isTransientDBError(err error) bool {\n\tvar sqliteErr sqlite3.Error\n\tif errors.As(err, &sqliteErr) && sqliteErr.Code == sqlite3.ErrBusy {\n\t\treturn true\n\t}\n\treturn false\n}","tryCatchPattern":"var nodes []string\nerr := retryOnTransient(5, func() error {\n\tvar e error\n\tnodes, e = db.BackfillNodeIPs(ipAlloc)\n\treturn e\n})\nif err != nil {\n\tlog.Fatal().Err(err).Msg(\"backfill failed after retries\")\n}","preventionTips":["Ensure a single headscale writer per SQLite file","Configure busy_timeout for SQLite","Run backfill at startup before accepting registrations"],"tags":["go","database","sqlite","postgresql","transaction"],"backgroundTag":null,"analyzedSha":"565fd254d06c4c7f9a8cad1714a43445c79ba420","analyzedAt":"2026-08-15T13:12:30.133Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}