crowdsecurity/crowdsec · error · UpdateFail

updating machine last_heartbeat: %w: %w

Error message

updating machine last_heartbeat: %w: %w

What it means

UpdateMachineLastHeartBeat wraps UpdateFail when setting last_heartbeat on a machine row fails. The heartbeat endpoint calls this on every watcher ping, so an error here means the DB rejected the write while a watcher was alive. Watchers will see their heartbeat request fail (usually surfaced as a 500/4xx by HandleDBErrors).

Source

Thrown at pkg/database/machines.go:195

func (c *Client) BulkDeleteWatchers(ctx context.Context, machines []*ent.Machine) (int, error) {
	ids := make([]int, len(machines))
	for i, b := range machines {
		ids[i] = b.ID
	}

	nbDeleted, err := c.Ent.Machine.Delete().Where(machine.IDIn(ids...)).Exec(ctx)
	if err != nil {
		return nbDeleted, err
	}

	return nbDeleted, nil
}

func (c *Client) UpdateMachineLastHeartBeat(ctx context.Context, machineID string) error {
	_, err := c.Ent.Machine.Update().Where(machine.MachineIdEQ(machineID)).SetLastHeartbeat(time.Now().UTC()).Save(ctx)
	if err != nil {
		return fmt.Errorf("updating machine last_heartbeat: %w: %w", err, UpdateFail)
	}

	return nil
}

func (c *Client) UpdateMachineScenarios(ctx context.Context, scenarios string, id int) error {
	_, err := c.Ent.Machine.UpdateOneID(id).
		SetUpdatedAt(time.Now().UTC()).
		SetScenarios(scenarios).
		Save(ctx)
	if err != nil {
		return fmt.Errorf("unable to update machine in database: %w", err)
	}

	return nil
}

func (c *Client) UpdateMachineIP(ctx context.Context, ipAddr string, id int) error {

View on GitHub (pinned to 909b515798)

Solutions

1. Check the wrapped error to identify the SQL cause
2. Confirm the DB is healthy: `cscli machines list` from the same host
3. Under sustained heartbeat load on SQLite, switch db_config to MySQL/Postgres
4. Restart LAPI to rebuild DB connection pools after a DB restart
5. Monitor disk space on the machine hosting the DB

// check machine resolvable before heartbeat persistence
if _, err := db.QueryMachineByID(ctx, machineID); err != nil { return err }

if err := db.UpdateMachineLastHeartBeat(ctx, machineID); err != nil {
    if errors.Is(err, database.UpdateFail) {
        // transient DB issue: return 503 to watcher so it retries heartbeat
    }
    return err
}

When it happens

Trigger: HeartBeat -> UpdateMachineLastHeartBeat with the DB down or the UPDATE failing: SQLite locked, connection pool exhausted, DB restart mid-request, disk full.

Common situations: Many watchers heartbeating at once overloading a slow SQLite file; DB maintenance/restart window; container OOM-killing the DB; wrong db_config after host migration.

Related errors

• validating machine: %w: %w
• unable to update machine in database: %w
• unable to update machine IP in database: %w
• unable to update machine version in database: %w
• no database configuration provided

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/bd94e630fbed9ae9. Report an issue: GitHub.