juanfont/headscale · error

listing nodes to backfill IPs: %w

Error message

listing nodes to backfill IPs: %w

What it means

Inside the backfill write transaction, ListNodes loads every node to decide which need IPs added or removed. This error means the SELECT itself failed at the database layer; the message only adds context, the underlying error (lock timeout, lost connection, corrupt table) is in %w. Because it runs inside db.Write, the transaction rolls back cleanly.

Source

Thrown at hscontrol/db/ip.go:321

// it will be added.
// If a prefix type has been removed (IPv4 or IPv6), it
// will remove the IPs in that family from the node.
func (db *HSDatabase) BackfillNodeIPs(i *IPAllocator) ([]string, error) {
	var (
		err error
		ret []string
	)

	err = db.Write(func(tx *gorm.DB) error {
		if i == nil {
			return fmt.Errorf("backfilling IPs: %w", errIPAllocatorNil)
		}

		log.Trace().Caller().Msgf("starting to backfill IPs")

		nodes, err := ListNodes(tx)
		if err != nil {
			return fmt.Errorf("listing nodes to backfill IPs: %w", err)
		}

		for _, node := range nodes {
			log.Trace().Caller().EmbedObject(node).Msg("ip backfill check started because node found in database")

			changed := false
			// IPv4 prefix is set, but node ip is missing, alloc
			if i.prefix4 != nil && node.IPv4 == nil {
				ret4, err := i.allocateNext(&i.prev4, i.prefix4)
				if err != nil {
					return fmt.Errorf("allocating IPv4 for node(%d): %w", node.ID, err)
				}

				node.IPv4 = ret4
				changed = true

				ret = append(ret, fmt.Sprintf("assigned IPv4 %q to Node(%d) %q", ret4.String(), node.ID, node.Hostname))
			}

View on GitHub (pinned to 565fd254d0)

Solutions

1. Check the wrapped error with errors.Unwrap to identify lock vs connectivity vs corruption
2. For SQLite locking, ensure only one headscale process uses the file and busy_timeout is configured
3. Verify disk space and database integrity (sqlite3 PRAGMA integrity_check / pg connectivity)
4. Re-run startup once the underlying database issue is fixed; backfill is idempotent

// Verify DB reachability before startup backfill
if err := db.Ping(); err != nil {
	return fmt.Errorf("database unreachable before backfill: %w", err)
}

func isTransientDBError(err error) bool {
	var sqliteErr sqlite3.Error
	if errors.As(err, &sqliteErr) && sqliteErr.Code == sqlite3.ErrBusy {
		return true
	}
	return false
}

var nodes []string
err := retryOnTransient(5, func() error {
	var e error
	nodes, e = db.BackfillNodeIPs(ipAlloc)
	return e
})
if err != nil {
	log.Fatal().Err(err).Msg("backfill failed after retries")
}

When it happens

Trigger: SQLite 'database is locked' from a concurrent writer; PostgreSQL connection dropped mid-transaction; nodes table corruption; a migration that has not run leaving the table in an unexpected shape.

Common situations: Running another headscale instance against the same SQLite file; disk-full on the database volume; a crash mid-upgrade leaving a stale -wal/-shm file; Postgres restarted under load.

Related errors

• saving node(%d) after adding IPs: %w
• checking name uniqueness: %w
• foreign key constraints violated
• node not found
• path cannot be empty

AI-assisted analysis of juanfont/headscale@565fd254d0 (2026-08-15). Data as JSON: /api/errors/8d7096e9f15fbcc7. Report an issue: GitHub.