k3s-io/k3s · error

%s disarm failed: %v

Error message

%s disarm failed: %v

What it means

When clearAlarms finds a NOSPACE alarm on the local member it attempts AlarmsDisarm to clear it after (per surrounding logic) space has been reclaimed. If the disarm RPC fails, this error reports the alarm type and the clientv3 error. The alarm remains armed and etcd may still reject writes.

Source

Thrown at pkg/etcd/etcd.go:1434

// Other alarm types are not handled.
func (e *ETCD) clearAlarms(ctx context.Context, memberID uint64) error {
	if e.client == nil {
		return errors.New("etcd client was nil")
	}

	alarmList, err := e.client.AlarmList(ctx)
	if err != nil {
		return fmt.Errorf("etcd alarm list failed: %v", err)
	}

	for _, alarm := range alarmList.Alarms {
		if alarm.MemberID != memberID {
			// ignore alarms on other cluster members, they should manage their own problems
			continue
		}
		if alarm.Alarm == etcdserverpb.AlarmType_NOSPACE {
			if _, err := e.client.AlarmDisarm(ctx, &clientv3.AlarmMember{MemberID: alarm.MemberID, Alarm: alarm.Alarm}); err != nil {
				return fmt.Errorf("%s disarm failed: %v", alarm.Alarm, err)
			}
			logrus.Infof("%s disarmed successfully", alarm.Alarm)
		} else {
			return fmt.Errorf("%s alarm must be disarmed manually", alarm.Alarm)
		}
	}
	return nil
}

// status returns status using the first etcd endpoint.
func (e *ETCD) status(ctx context.Context) (*clientv3.StatusResponse, error) {
	if e.client == nil {
		return nil, errors.New("etcd client was nil")
	}

	ctx, cancel := context.WithTimeout(ctx, statusTimeout)
	defer cancel()

View on GitHub (pinned to 6ba341e396)

Solutions

1. Free real space AND check the quota: if --etcd-quota-backend-bytes is set, ensure the backend size is below it (compact/defrag: `etcdctl compact` + `etcdctl defrag`) before retrying.
2. Disarm manually once the member is healthy: `ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=... --cert=... --key=... alarm disarm`.
3. If disarm keeps failing, restart the etcd member (restart k3s server) so the alarm sweep re-runs against a fresh connection.

Example fix

# after (manual disarm once healthy)
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
  --cert=/var/lib/rancher/k3s/server/tls/etcd/client.crt \
  --key=/var/lib/rancher/k3s/server/tls/etcd/client.key \
  alarm disarm

// Confirm space was actually reclaimed before disarm: backend size must be under quota:
st, err := cli.Status(ctx, endpoint)
if err == nil && quotaBytes > 0 && st.DbSize > quotaBytes {
    log.Printf("backend %d still over quota %d - defrag first", st.DbSize, quotaBytes)
}

// Retry disarm after remediation, give up and surface after N attempts:
err := retry(3, 5*time.Second, func() error {
    _, err := cli.AlarmDisarm(ctx, &clientv3.AlarmMember{MemberID: id, Alarm: etcdserverpb.AlarmType_NOSPACE})
    return err
})
if err != nil { log.Fatalf("disarm failed; run etcdctl alarm disarm manually once healthy: %v", err) }

When it happens

Trigger: e.client.AlarmDisarm(...) returning an error while disarming a NOSPACE alarm on the local member (pkg/etcd/etcd.go:1431-1435): the member went unresponsive mid-call, the backend is still over quota so etcd refuses to disarm, or connection/TLS issues.

Common situations: Disk freed but quota-backend-bytes limit still exceeded so disarm is rejected; etcd restarted or crashed between alarm list and disarm; slow disk causing gRPC timeout.

Related errors

• etcd alarm list failed: %v
• %s alarm must be disarmed manually
• no snapshots given for removal
• invalid output format:
• etcd-snapshot-reconcile-interval must be greater than 0s

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/737575e7ccdcea78. Report an issue: GitHub.