k3s-io/k3s · error

etcd alarm list failed: %v

Error message

etcd alarm list failed: %v

What it means

clearAlarms queries the local etcd member for active alarms (e.g. NOSPACE raised when the backend quota is exceeded). If the AlarmList RPC itself fails - client cannot reach etcd, timeout, member not serving - this error wraps the clientv3 failure. A nil client is a separate error ('etcd client was nil').

Source

Thrown at pkg/etcd/etcd.go:1424

	if err := json.NewEncoder(w).Encode(status); err != nil {
		return err
	}

	_, err := e.client.Put(ctx, learnerProgressKey, w.String())
	return err
}

// clearAlarms checks for any NOSPACE alarms on the local etcd member.
// If found, they are reported and the alarm state is cleared.
// Other alarm types are not handled.
func (e *ETCD) clearAlarms(ctx context.Context, memberID uint64) error {
	if e.client == nil {
		return errors.New("etcd client was nil")
	}

	alarmList, err := e.client.AlarmList(ctx)
	if err != nil {
		return fmt.Errorf("etcd alarm list failed: %v", err)
	}

	for _, alarm := range alarmList.Alarms {
		if alarm.MemberID != memberID {
			// ignore alarms on other cluster members, they should manage their own problems
			continue
		}
		if alarm.Alarm == etcdserverpb.AlarmType_NOSPACE {
			if _, err := e.client.AlarmDisarm(ctx, &clientv3.AlarmMember{MemberID: alarm.MemberID, Alarm: alarm.Alarm}); err != nil {
				return fmt.Errorf("%s disarm failed: %v", alarm.Alarm, err)
			}
			logrus.Infof("%s disarmed successfully", alarm.Alarm)
		} else {
			return fmt.Errorf("%s alarm must be disarmed manually", alarm.Alarm)
		}
	}
	return nil
}

View on GitHub (pinned to 6ba341e396)

Solutions

1. Check etcd member health: `k3s etcd-snapshot check --latest` or etcdctl endpoint status; look at the etcd log lines around the failure.
2. Address the resource problem first - NOSPACE incidents usually mean the disk/quota is exhausted: free space or raise --etcd-quota-backend-bytes, then restart k3s.
3. Verify disk space (df -h), inodes (df -i) and the data-dir filesystem before retrying startup.

// Verify the local etcd endpoint answers before the operation that clears alarms:
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if _, err := clientv3.New(clientv3.Config{Endpoints: []string{"https://127.0.0.1:2379"}, TLS: tlsCfg}).Status(ctx, "https://127.0.0.1:2379"); err != nil {
    log.Fatalf("etcd not ready: %v", err)
}

// Retry the alarm list with backoff while etcd is settling:
var alarms *clientv3.AlarmResponse
err := retry(5, 2*time.Second, func() error {
    var err error
    alarms, err = cli.AlarmList(ctx)
    return err // retry on transient unavailability; abort on auth errors
})
if err != nil { /* etcd still down - investigate health, do not loop forever */ }

When it happens

Trigger: e.client.AlarmList(ctx) failing during the post-start alarm sweep (pkg/etcd/etcd.go:1423-1425): local etcd endpoint down or not yet listening, gRPC deadline exceeded, TLS/auth mismatch, or etcd crashed (often itself a symptom of disk-full NOSPACE).

Common situations: Disk full on an etcd node making etcd unresponsive; etcd still starting up when the alarm check ran; wrong client cert/ETCDCTL_ENDPOINTS; frequent during the same incident that raised the alarms.

Related errors

• %s disarm failed: %v
• %s alarm must be disarmed manually
• no snapshots given for removal
• invalid output format:
• etcd-snapshot-reconcile-interval must be greater than 0s

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/66ed1ea61ee7361d. Report an issue: GitHub.